Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250904 01:54:52.628127 3380 test_util.cc:276] Using random seed: 30918781
2025-09-04T01:54:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T01:54:52Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskHkVPgV/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskHkVPgV/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/security.properties
2025-09-04 01:54:58: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskHkVPgV/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskHkVPgV/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250904 01:55:05.525591 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--webserver_interface=127.3.77.62
--webserver_port=0
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:05.806631 3621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:05.806867 3621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:05.806990 3621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:05.814265 3621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:05.814332 3621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:05.814394 3621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:05.814488 3621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:05.822824 3621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.77.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:05.824124 3621 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:05.825595 3621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:05.832274 3626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:05.832309 3627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:05.832782 3629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:05.833174 3621 server_base.cc:1047] running on GCE node
I20250904 01:55:05.833808 3621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:05.835000 3621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:05.836181 3621 hybrid_clock.cc:648] HybridClock initialized: now 1756950905836142 us; error 32 us; skew 500 ppm
I20250904 01:55:05.836458 3621 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:05.838181 3621 webserver.cc:480] Webserver started at http://127.3.77.62:46147/ using document root <none> and password file <none>
I20250904 01:55:05.838883 3621 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:05.838974 3621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:05.839313 3621 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:55:05.841274 3621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/instance:
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:05.841950 3621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal/instance:
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:05.846486 3621 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250904 01:55:05.849282 3635 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:05.850329 3621 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250904 01:55:05.850471 3621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:05.850623 3621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:05.867298 3621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:05.867918 3621 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:05.868137 3621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:05.883981 3621 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.62:34273
I20250904 01:55:05.883999 3686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.62:34273 every 8 connection(s)
I20250904 01:55:05.885130 3621 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
I20250904 01:55:05.888160 3687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:05.890017 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 3621
I20250904 01:55:05.890341 3380 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal/instance
I20250904 01:55:05.893582 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--webserver_interface=127.3.77.61
--webserver_port=0
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 01:55:05.896003 3687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:55:05.909906 3687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:55:05.910882 3637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:05.912204 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:05.961501 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } attempt: 1
W20250904 01:55:05.963136 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:06.031376 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } attempt: 2
W20250904 01:55:06.033046 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:06.124301 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } attempt: 3
W20250904 01:55:06.125847 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:06.174875 3688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:06.175104 3688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:06.175181 3688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:06.182200 3688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:06.182281 3688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:06.182328 3688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:06.182374 3688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:06.190569 3688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.77.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:06.191730 3688 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:06.193147 3688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:06.200089 3697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:06.200130 3696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:06.200858 3699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:06.201275 3688 server_base.cc:1047] running on GCE node
I20250904 01:55:06.201889 3688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:06.203104 3688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:06.204283 3688 hybrid_clock.cc:648] HybridClock initialized: now 1756950906204252 us; error 29 us; skew 500 ppm
I20250904 01:55:06.204555 3688 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:06.206995 3688 webserver.cc:480] Webserver started at http://127.3.77.61:46249/ using document root <none> and password file <none>
I20250904 01:55:06.207567 3688 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:06.207652 3688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:06.207940 3688 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:55:06.211099 3688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/instance:
uuid: "6ef804bcecd943d89c7af93e9375b1f3"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:06.211740 3688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal/instance:
uuid: "6ef804bcecd943d89c7af93e9375b1f3"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:06.216114 3688 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250904 01:55:06.218976 3705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:06.220041 3688 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250904 01:55:06.220181 3688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
uuid: "6ef804bcecd943d89c7af93e9375b1f3"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:06.220346 3688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:06.242810 3688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:06.243486 3688 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:06.243711 3688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:06.260461 3688 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.61:36307
I20250904 01:55:06.260468 3756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.61:36307 every 8 connection(s)
I20250904 01:55:06.261620 3688 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
I20250904 01:55:06.264698 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:06.268354 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 3688
I20250904 01:55:06.268535 3380 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal/instance
I20250904 01:55:06.269073 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } attempt: 4
I20250904 01:55:06.269778 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.60:38761
--webserver_interface=127.3.77.60
--webserver_port=0
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 01:55:06.270764 3757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:55:06.284116 3757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:55:06.285147 3687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:55:06.286935 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:06.290506 3757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:55:06.291316 3706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:06.292758 3757 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:06.342002 3757 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 1
W20250904 01:55:06.343751 3757 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:06.346132 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 1
W20250904 01:55:06.347640 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:06.412048 3757 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 2
W20250904 01:55:06.413841 3757 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:06.414829 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 2
W20250904 01:55:06.416466 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:06.505019 3757 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 3
W20250904 01:55:06.506799 3757 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:06.516760 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 3
W20250904 01:55:06.518291 3687 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:06.567399 3759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:06.567665 3759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:06.567771 3759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:06.575457 3759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:06.575551 3759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:06.575656 3759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:06.575713 3759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:06.584046 3759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.60:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.77.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:06.585376 3759 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:06.586977 3759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:06.594019 3766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:06.593983 3767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:06.594678 3769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:06.594942 3759 server_base.cc:1047] running on GCE node
I20250904 01:55:06.595561 3759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:06.596685 3759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:06.597873 3759 hybrid_clock.cc:648] HybridClock initialized: now 1756950906597829 us; error 39 us; skew 500 ppm
I20250904 01:55:06.598145 3759 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:06.600114 3759 webserver.cc:480] Webserver started at http://127.3.77.60:41119/ using document root <none> and password file <none>
I20250904 01:55:06.600780 3759 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:06.600876 3759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:06.601202 3759 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:55:06.603397 3759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/instance:
uuid: "7f298ba82a5e49028407388810de4b71"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:06.604116 3759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal/instance:
uuid: "7f298ba82a5e49028407388810de4b71"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:06.608371 3759 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250904 01:55:06.611155 3775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:06.612241 3759 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250904 01:55:06.612387 3759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
uuid: "7f298ba82a5e49028407388810de4b71"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:06.612576 3759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:06.622977 3759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:06.623641 3759 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:06.623879 3759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:06.641906 3759 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.60:38761
I20250904 01:55:06.641927 3826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.60:38761 every 8 connection(s)
I20250904 01:55:06.643177 3759 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/info.pb
I20250904 01:55:06.646220 3827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:06.646978 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 3759
I20250904 01:55:06.647178 3380 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal/instance
I20250904 01:55:06.650032 3757 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 4
I20250904 01:55:06.653915 3827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:55:06.672617 3757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap starting.
I20250904 01:55:06.672863 3827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:55:06.677852 3757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:06.679478 3757 log.cc:826] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:06.683532 3757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: No bootstrap required, opened a new log
I20250904 01:55:06.683928 3827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:55:06.688499 3687 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } attempt: 4
I20250904 01:55:06.693245 3757 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:06.693572 3757 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:06.693678 3757 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef804bcecd943d89c7af93e9375b1f3, State: Initialized, Role: FOLLOWER
I20250904 01:55:06.695453 3757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:06.699658 3757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:06.700585 3827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Bootstrap starting.
I20250904 01:55:06.698495 3835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:06.701124 3835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:06.710906 3687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap starting.
I20250904 01:55:06.712884 3827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:06.714094 3827 log.cc:826] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:06.714486 3687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:06.715835 3687 log.cc:826] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:06.721066 3827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: No bootstrap required, opened a new log
I20250904 01:55:06.732535 3827 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:06.732918 3827 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:06.733023 3827 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f298ba82a5e49028407388810de4b71, State: Initialized, Role: FOLLOWER
I20250904 01:55:06.725508 3688 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: OK, instance_id { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" instance_seqno: 1756950905879893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:55:06.733493 3688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:06.733754 3827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
W20250904 01:55:06.736210 3851 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:55:06.736354 3851 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 01:55:06.736651 3687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: No bootstrap required, opened a new log
W20250904 01:55:06.745434 3711 tablet.cc:2378] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:55:06.745599 3687 raft_consensus.cc:357] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:06.745891 3687 raft_consensus.cc:383] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:06.745982 3687 raft_consensus.cc:738] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Initialized, Role: FOLLOWER
I20250904 01:55:06.746790 3687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:06.747875 3854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:06.748160 3854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:06.749286 3827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:06.758744 3855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:06.759074 3855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:06.763736 3687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: configured and running, proceeding with master startup.
W20250904 01:55:06.870489 3641 tablet.cc:2378] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:55:06.871686 3879 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250904 01:55:06.875748 3781 tablet.cc:2378] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:55:06.881021 3621 master_runner.cc:186] Error getting master registration for 127.3.77.60:38761: OK, instance_id { permanent_uuid: "7f298ba82a5e49028407388810de4b71" instance_seqno: 1756950906637187 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:55:06.895293 3621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:06.915256 3759 master_runner.cc:263] No leader master found from master 7f298ba82a5e49028407388810de4b71
I20250904 01:55:06.915367 3759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:55:06.920873 3885 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:55:06.920995 3885 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 01:55:06.986835 3855 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:06.987064 3855 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:06.988301 3855 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:06.988910 3732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:06.989058 3802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f298ba82a5e49028407388810de4b71" is_pre_election: true
I20250904 01:55:06.989377 3732 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 0.
I20250904 01:55:06.989547 3802 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 0.
I20250904 01:55:06.990061 3636 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71; no voters:
I20250904 01:55:06.990664 3855 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:06.990796 3855 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:06.990885 3855 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:06.993726 3855 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:06.994727 3732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3"
I20250904 01:55:06.994832 3802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f298ba82a5e49028407388810de4b71"
I20250904 01:55:06.994903 3732 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:06.994992 3802 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:06.997994 3802 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 1.
I20250904 01:55:06.998097 3732 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 1.
I20250904 01:55:06.998337 3636 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71; no voters:
I20250904 01:55:06.998678 3855 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 1 election: Requested vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:06.998890 3855 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:06.999044 3855 raft_consensus.cc:695] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 1 LEADER]: Becoming Leader. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Running, Role: LEADER
I20250904 01:55:06.999497 3855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:07.013090 3887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 1 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:07.013579 3887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:07.014335 3889 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:07.019240 3889 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:07.032496 3732 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Refusing update from remote peer 29b8a62cc8d04df781f991da2bd83ccc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:55:07.033461 3802 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Refusing update from remote peer 29b8a62cc8d04df781f991da2bd83ccc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:55:07.034303 3855 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:07.034544 3887 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:07.060067 3835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 1 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:07.060259 3835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:07.062487 3891 mvcc.cc:204] Tried to move back new op lower bound from 7196470915195858944 to 7196470915074994176. Current Snapshot: MvccSnapshot[applied={T|T < 7196470915195858944}]
I20250904 01:55:07.064260 3887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:07.064445 3887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:07.068166 3889 catalog_manager.cc:1349] Generated new cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:07.068276 3889 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:07.083947 3835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:07.084180 3835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:07.093371 3889 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 01:55:07.094993 3889 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:07.105106 3854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 1 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:07.105343 3854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:07.107553 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:07.107795 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:07.118696 3895 mvcc.cc:204] Tried to move back new op lower bound from 7196470915195858944 to 7196470915074994176. Current Snapshot: MvccSnapshot[applied={T|T < 7196470915195858944}]
I20250904 01:55:07.119499 3854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:07.120922 3854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:07.131678 3889 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Generated new TSK 0
I20250904 01:55:07.132550 3889 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:07.133788 3889 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:07.135033 3889 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:07.173223 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.77.1:0
--local_ip_for_outbound_sockets=127.3.77.1
--webserver_interface=127.3.77.1
--webserver_port=0
--tserver_master_addrs=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250904 01:55:07.560724 3900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:07.560949 3900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:07.561029 3900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:07.568343 3900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:07.568482 3900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.77.1
I20250904 01:55:07.576943 3900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.77.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.77.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.77.1
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:07.578164 3900 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:07.579684 3900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:07.588236 3906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:07.588296 3905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:07.591212 3908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:07.591470 3900 server_base.cc:1047] running on GCE node
I20250904 01:55:07.591941 3900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:07.592687 3900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:07.593875 3900 hybrid_clock.cc:648] HybridClock initialized: now 1756950907593835 us; error 35 us; skew 500 ppm
I20250904 01:55:07.594132 3900 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:07.596060 3900 webserver.cc:480] Webserver started at http://127.3.77.1:32929/ using document root <none> and password file <none>
I20250904 01:55:07.596670 3900 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:07.596762 3900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:07.597100 3900 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:55:07.599197 3900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data/instance:
uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
format_stamp: "Formatted at 2025-09-04 01:55:07 on dist-test-slave-ch6q"
I20250904 01:55:07.599858 3900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/instance:
uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
format_stamp: "Formatted at 2025-09-04 01:55:07 on dist-test-slave-ch6q"
I20250904 01:55:07.604487 3900 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250904 01:55:07.607198 3914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:07.608208 3900 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250904 01:55:07.608341 3900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal
uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
format_stamp: "Formatted at 2025-09-04 01:55:07 on dist-test-slave-ch6q"
I20250904 01:55:07.608505 3900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:07.633015 3919 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:07.633199 3919 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:07.634413 3919 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
I20250904 01:55:07.634900 3662 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
I20250904 01:55:07.634912 3732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:07.635623 3776 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f298ba82a5e49028407388810de4b71; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:07.635926 3919 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:55:07.639554 3900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:07.640161 3900 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:07.640376 3900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:07.641142 3900 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:55:07.642654 3900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:55:07.642745 3900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:07.642848 3900 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:55:07.642916 3900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:07.673452 3900 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.1:37169
I20250904 01:55:07.673544 4027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.1:37169 every 8 connection(s)
I20250904 01:55:07.674753 3900 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/data/info.pb
I20250904 01:55:07.682284 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 3900
I20250904 01:55:07.682515 3380 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/instance
I20250904 01:55:07.687363 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.77.2:0
--local_ip_for_outbound_sockets=127.3.77.2
--webserver_interface=127.3.77.2
--webserver_port=0
--tserver_master_addrs=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:55:07.718041 4029 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:07.718456 4029 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:07.718991 4028 heartbeater.cc:344] Connected to a master server at 127.3.77.60:38761
I20250904 01:55:07.719159 4028 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:07.719522 4029 heartbeater.cc:507] Master 127.3.77.62:34273 requested a full tablet report, sending...
I20250904 01:55:07.719775 4028 heartbeater.cc:507] Master 127.3.77.60:38761 requested a full tablet report, sending...
I20250904 01:55:07.721500 3792 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:07.721518 3652 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:07.721963 4030 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:07.722074 4030 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:07.722575 4030 heartbeater.cc:507] Master 127.3.77.61:36307 requested a full tablet report, sending...
I20250904 01:55:07.723412 3652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.77.1:37387
I20250904 01:55:07.724269 3722 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:07.737892 3851 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:07.737962 3851 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: loading cluster ID for follower catalog manager: success
I20250904 01:55:07.740161 3851 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: acquiring CA information for follower catalog manager: success
I20250904 01:55:07.741672 3851 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:07.922824 3885 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:07.922909 3885 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: loading cluster ID for follower catalog manager: success
I20250904 01:55:07.925417 3885 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: acquiring CA information for follower catalog manager: success
I20250904 01:55:07.926996 3885 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250904 01:55:08.012442 4036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:08.012696 4036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:08.012815 4036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:08.020388 4036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:08.020582 4036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.77.2
I20250904 01:55:08.029254 4036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.77.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.77.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.77.2
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:08.030656 4036 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:08.032173 4036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:08.040855 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:08.041005 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:08.041111 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:08.042150 4036 server_base.cc:1047] running on GCE node
I20250904 01:55:08.042680 4036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:08.043475 4036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:08.044660 4036 hybrid_clock.cc:648] HybridClock initialized: now 1756950908044613 us; error 38 us; skew 500 ppm
I20250904 01:55:08.044924 4036 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:08.046837 4036 webserver.cc:480] Webserver started at http://127.3.77.2:35541/ using document root <none> and password file <none>
I20250904 01:55:08.047485 4036 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:08.047587 4036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:08.047907 4036 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:55:08.050029 4036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data/instance:
uuid: "5b23e87ab60248f391fb17326f5e3934"
format_stamp: "Formatted at 2025-09-04 01:55:08 on dist-test-slave-ch6q"
I20250904 01:55:08.050770 4036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/instance:
uuid: "5b23e87ab60248f391fb17326f5e3934"
format_stamp: "Formatted at 2025-09-04 01:55:08 on dist-test-slave-ch6q"
I20250904 01:55:08.055359 4036 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250904 01:55:08.058410 4052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:08.059531 4036 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250904 01:55:08.059695 4036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal
uuid: "5b23e87ab60248f391fb17326f5e3934"
format_stamp: "Formatted at 2025-09-04 01:55:08 on dist-test-slave-ch6q"
I20250904 01:55:08.059883 4036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:08.076378 4036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:08.077076 4036 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:08.077320 4036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:08.078120 4036 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:55:08.079502 4036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:55:08.079598 4036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:08.079715 4036 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:55:08.079787 4036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:08.099722 4131 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:08.099905 4131 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:08.101226 4131 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:08.101749 3662 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
I20250904 01:55:08.101855 3801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7f298ba82a5e49028407388810de4b71" is_pre_election: true
I20250904 01:55:08.102598 3706 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:08.102967 4131 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:55:08.110407 4036 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.2:40911
I20250904 01:55:08.110524 4165 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.2:40911 every 8 connection(s)
I20250904 01:55:08.111723 4036 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/data/info.pb
I20250904 01:55:08.120870 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 4036
I20250904 01:55:08.121057 3380 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/instance
I20250904 01:55:08.124689 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.77.3:0
--local_ip_for_outbound_sockets=127.3.77.3
--webserver_interface=127.3.77.3
--webserver_port=0
--tserver_master_addrs=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:55:08.130064 4166 heartbeater.cc:344] Connected to a master server at 127.3.77.60:38761
I20250904 01:55:08.130450 4166 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:08.131251 4167 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:08.131397 4167 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:08.131466 4166 heartbeater.cc:507] Master 127.3.77.60:38761 requested a full tablet report, sending...
I20250904 01:55:08.131940 4167 heartbeater.cc:507] Master 127.3.77.62:34273 requested a full tablet report, sending...
I20250904 01:55:08.133186 3792 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:08.133246 3652 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:08.133913 4168 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:08.134089 4168 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:08.134174 3652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.77.2:60709
I20250904 01:55:08.134720 4168 heartbeater.cc:507] Master 127.3.77.61:36307 requested a full tablet report, sending...
I20250904 01:55:08.135741 3722 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:08.180297 4174 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:08.180454 4174 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:08.181066 4174 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
I20250904 01:55:08.181504 3732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:08.181561 3662 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
I20250904 01:55:08.182116 3776 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f298ba82a5e49028407388810de4b71; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:08.182382 4174 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:55:08.225703 4029 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
W20250904 01:55:08.420696 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:08.420908 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:08.420976 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:08.428210 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:08.428341 4173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.77.3
I20250904 01:55:08.436637 4173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.77.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.77.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.77.3
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:08.437861 4173 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:08.439360 4173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:08.447654 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:08.447633 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:08.447875 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:08.448473 4173 server_base.cc:1047] running on GCE node
I20250904 01:55:08.448928 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:08.449695 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:08.450860 4173 hybrid_clock.cc:648] HybridClock initialized: now 1756950908450820 us; error 38 us; skew 500 ppm
I20250904 01:55:08.451081 4173 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:08.452733 4173 webserver.cc:480] Webserver started at http://127.3.77.3:32779/ using document root <none> and password file <none>
I20250904 01:55:08.453333 4173 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:08.453410 4173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:08.453680 4173 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:55:08.455782 4173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data/instance:
uuid: "7d96d300478849e3b11402fbd34fef15"
format_stamp: "Formatted at 2025-09-04 01:55:08 on dist-test-slave-ch6q"
I20250904 01:55:08.456384 4173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/instance:
uuid: "7d96d300478849e3b11402fbd34fef15"
format_stamp: "Formatted at 2025-09-04 01:55:08 on dist-test-slave-ch6q"
I20250904 01:55:08.460692 4173 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20250904 01:55:08.463487 4188 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:08.464526 4173 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250904 01:55:08.464673 4173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal
uuid: "7d96d300478849e3b11402fbd34fef15"
format_stamp: "Formatted at 2025-09-04 01:55:08 on dist-test-slave-ch6q"
I20250904 01:55:08.464839 4173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:08.480216 4173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:08.480804 4173 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:08.481045 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:08.481765 4173 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:55:08.483196 4173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:55:08.483278 4173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:08.483387 4173 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:55:08.483462 4173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:08.512243 4173 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.3:39819
I20250904 01:55:08.512306 4300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.3:39819 every 8 connection(s)
I20250904 01:55:08.513495 4173 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/data/info.pb
I20250904 01:55:08.521863 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 4173
I20250904 01:55:08.522048 3380 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/instance
I20250904 01:55:08.528368 4301 heartbeater.cc:344] Connected to a master server at 127.3.77.60:38761
I20250904 01:55:08.528491 4302 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:08.528745 4301 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:08.528751 4302 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:08.529564 4301 heartbeater.cc:507] Master 127.3.77.60:38761 requested a full tablet report, sending...
I20250904 01:55:08.529587 4302 heartbeater.cc:507] Master 127.3.77.62:34273 requested a full tablet report, sending...
I20250904 01:55:08.528344 4303 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:08.530313 4303 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:08.530759 4303 heartbeater.cc:507] Master 127.3.77.61:36307 requested a full tablet report, sending...
I20250904 01:55:08.530936 3652 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:08.531049 3792 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:08.531631 3722 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:08.531849 3652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.77.3:36547
I20250904 01:55:08.546047 3380 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 01:55:08.570739 3652 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_ac1709b4bdce4074b1fac1b503f0d518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.571293 3650 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.571466 3649 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_d150897ff208409984d49a17e902c6c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.570785 3651 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_a524c1ec64554d8eb7687bfa84e4579c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.574121 3647 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_18be6a726bc64f8c8efad17c700b66aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.575033 3648 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_5a0ac2833a3d4a2b98f538f7416ca2d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.575687 3646 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.576783 3645 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_7d5858a140ad4b7eb83927265abad9fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.577574 3644 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_9a196bab16bb4604ab9b00d45335f022"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.578534 3643 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46788:
name: "default.Table_e916036e5adf47c886c74e8088132031"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 01:55:08.579216 3652 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ac1709b4bdce4074b1fac1b503f0d518 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:08.581768 3650 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a89876a7c4d946f09bfa330cab1f34f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:08.584853 3649 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d150897ff208409984d49a17e902c6c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:08.585973 3651 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a524c1ec64554d8eb7687bfa84e4579c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:08.587085 3647 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_18be6a726bc64f8c8efad17c700b66aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:08.588245 3648 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:08.588683 3645 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7d5858a140ad4b7eb83927265abad9fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:08.589862 3646 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9c770f9d8cf94c06a272116f40e60c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 01:55:08.597436 3380 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu pid 3621
I20250904 01:55:08.598668 3682 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 01:55:08.636207 4167 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
I20250904 01:55:08.688647 4348 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:08.688966 4348 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:08.689749 4348 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:08.690340 3801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7f298ba82a5e49028407388810de4b71" is_pre_election: true
I20250904 01:55:08.746179 3380 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu with pid 3621
W20250904 01:55:08.772009 4308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:08.776574 4167 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.3.77.62:34273 (error 108)
W20250904 01:55:08.776665 4029 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.3.77.62:34273 (error 108)
W20250904 01:55:08.785180 4313 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:08.773430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.62:34273 (local address 127.0.0.1:46790)
0904 01:55:08.774920 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:08.774942 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:08.775114 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:08.783673 (+ 8559us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1242}
I20250904 01:55:08.789013 4350 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:08.789634 4350 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
W20250904 01:55:08.794342 3776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:55:08.795944 4350 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
I20250904 01:55:08.796126 3732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:08.796326 3732 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f298ba82a5e49028407388810de4b71 in term 1.
I20250904 01:55:08.796826 3777 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71; no voters:
I20250904 01:55:08.797192 4350 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 01:55:08.797389 4350 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:08.797560 4350 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:55:08.800484 4350 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:08.802496 3732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3"
I20250904 01:55:08.802661 3732 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:55:08.805477 3732 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f298ba82a5e49028407388810de4b71 in term 2.
I20250904 01:55:08.805941 3777 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71; no voters:
I20250904 01:55:08.806514 4350 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 election: Requested vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
I20250904 01:55:08.806597 4354 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 2 FOLLOWER]: Leader election won for term 2
W20250904 01:55:08.806908 3706 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:55:08.807018 3706 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:08.807421 4348 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:55:08.806821 4354 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 2 LEADER]: Becoming Leader. State: Replica: 7f298ba82a5e49028407388810de4b71, State: Running, Role: LEADER
W20250904 01:55:08.809484 3776 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:55:08.809427 4354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
W20250904 01:55:08.810721 3776 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:55:08.820887 4350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f298ba82a5e49028407388810de4b71. Latest consensus state: current_term: 2 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:08.821080 4350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:08.829953 4356 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:08.832373 4356 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:08.833276 4356 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:08.833359 4356 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:08.834573 4356 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:08.835597 4356 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Loaded TSK: 0
I20250904 01:55:08.836690 4356 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:08.837575 4356 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:08.838476 4356 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:08.842975 3791 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_9a196bab16bb4604ab9b00d45335f022"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.843190 3788 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_a524c1ec64554d8eb7687bfa84e4579c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.844475 3784 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_5a0ac2833a3d4a2b98f538f7416ca2d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.844290 3783 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_18be6a726bc64f8c8efad17c700b66aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.844141 3786 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.844051 3785 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_d150897ff208409984d49a17e902c6c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.843770 3787 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_ac1709b4bdce4074b1fac1b503f0d518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.843549 3790 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_7d5858a140ad4b7eb83927265abad9fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.843350 3789 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:08.842898 3792 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_e916036e5adf47c886c74e8088132031"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 01:55:08.856844 3791 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9a196bab16bb4604ab9b00d45335f022 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 01:55:09.033749 4302 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
W20250904 01:55:09.036268 4302 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:55:09.072849 3791 catalog_manager.cc:2149] created HMS catalog entry for table [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:09.077071 3732 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Refusing update from remote peer 7f298ba82a5e49028407388810de4b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250904 01:55:09.077840 4350 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250904 01:55:09.080554 3776 consensus_peers.cc:489] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 -> Peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Couldn't send request to peer 29b8a62cc8d04df781f991da2bd83ccc. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 01:55:09.081725 3792 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e916036e5adf47c886c74e8088132031 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.081735 3788 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a524c1ec64554d8eb7687bfa84e4579c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.082036 3784 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.083421 3783 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_18be6a726bc64f8c8efad17c700b66aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.084683 3786 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a89876a7c4d946f09bfa330cab1f34f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.085760 3785 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d150897ff208409984d49a17e902c6c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.086570 3789 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9c770f9d8cf94c06a272116f40e60c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.087477 3790 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7d5858a140ad4b7eb83927265abad9fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 01:55:09.088292 3787 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ac1709b4bdce4074b1fac1b503f0d518 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 01:55:09.094520 4348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f298ba82a5e49028407388810de4b71. Latest consensus state: current_term: 2 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:09.094740 4348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:09.100664 4359 mvcc.cc:204] Tried to move back new op lower bound from 7196470923570966528 to 7196470922490241024. Current Snapshot: MvccSnapshot[applied={T|T < 7196470923570966528}]
I20250904 01:55:09.106523 4354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:09.106767 4354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:09.115295 3791 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_9a196bab16bb4604ab9b00d45335f022" } new_table_name: "default.Table_09793f14275347b0aaa65279c0b3bc41" modify_external_catalogs: true
I20250904 01:55:09.123175 4348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:09.123383 4348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:09.133410 3792 catalog_manager.cc:2149] created HMS catalog entry for table [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:09.184796 3792 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_e916036e5adf47c886c74e8088132031" } new_table_name: "default.Table_386e7bb279394e4ba14a48582a9ba8f8" modify_external_catalogs: true
I20250904 01:55:09.188953 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 735f984a7ed34c758c9032d28e3421ea (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.190316 3963 tablet_service.cc:1468] Processing CreateTablet for tablet 735f984a7ed34c758c9032d28e3421ea (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.190864 3962 tablet_service.cc:1468] Processing CreateTablet for tablet b30c0c07cbbf43d9aa616c7a025a0a91 (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.191756 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30c0c07cbbf43d9aa616c7a025a0a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.192138 3963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735f984a7ed34c758c9032d28e3421ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.190387 4099 tablet_service.cc:1468] Processing CreateTablet for tablet b30c0c07cbbf43d9aa616c7a025a0a91 (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.198930 4099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30c0c07cbbf43d9aa616c7a025a0a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.202425 3788 catalog_manager.cc:2149] created HMS catalog entry for table [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:09.202632 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735f984a7ed34c758c9032d28e3421ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.212811 4369 tablet_bootstrap.cc:492] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.213387 3788 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_a524c1ec64554d8eb7687bfa84e4579c" } new_table_name: "default.Table_3a626a942a5e41e9b5d523f0be9b7387" modify_external_catalogs: true
I20250904 01:55:09.215257 4236 tablet_service.cc:1468] Processing CreateTablet for tablet 735f984a7ed34c758c9032d28e3421ea (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.216591 4236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735f984a7ed34c758c9032d28e3421ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.216848 4235 tablet_service.cc:1468] Processing CreateTablet for tablet b30c0c07cbbf43d9aa616c7a025a0a91 (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.217442 4369 tablet_bootstrap.cc:654] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.217499 4235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30c0c07cbbf43d9aa616c7a025a0a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.218636 4369 log.cc:826] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:09.220332 4370 tablet_bootstrap.cc:492] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.221097 3962 tablet_service.cc:1468] Processing CreateTablet for tablet cdd783e309a44b039890d508fb577e9d (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.221218 4234 tablet_service.cc:1468] Processing CreateTablet for tablet cdd783e309a44b039890d508fb577e9d (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.221694 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd783e309a44b039890d508fb577e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.221731 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd783e309a44b039890d508fb577e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.222060 4100 tablet_service.cc:1468] Processing CreateTablet for tablet cdd783e309a44b039890d508fb577e9d (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.224182 4370 tablet_bootstrap.cc:654] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.224721 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd783e309a44b039890d508fb577e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.226027 4370 log.cc:826] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:09.228346 4369 tablet_bootstrap.cc:492] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.228608 4369 ts_tablet_manager.cc:1397] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20250904 01:55:09.237066 4369 raft_consensus.cc:357] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.237320 4369 raft_consensus.cc:383] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.237447 4369 raft_consensus.cc:738] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.237987 4369 consensus_queue.cc:260] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.240484 4370 tablet_bootstrap.cc:492] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.240767 4370 ts_tablet_manager.cc:1397] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20250904 01:55:09.243341 4369 ts_tablet_manager.cc:1428] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
I20250904 01:55:09.243827 4369 tablet_bootstrap.cc:492] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.245638 4369 tablet_bootstrap.cc:654] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.248370 4370 raft_consensus.cc:357] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.248776 4370 raft_consensus.cc:383] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.248904 4370 raft_consensus.cc:738] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.249703 4370 consensus_queue.cc:260] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.254853 4375 tablet_bootstrap.cc:492] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.258154 4375 tablet_bootstrap.cc:654] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.259712 4375 log.cc:826] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:09.261045 4369 tablet_bootstrap.cc:492] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.261045 4166 heartbeater.cc:499] Master 127.3.77.60:38761 was elected leader, sending a full tablet report...
I20250904 01:55:09.261238 4369 ts_tablet_manager.cc:1397] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.015s
I20250904 01:55:09.262034 4369 raft_consensus.cc:357] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.262293 4369 raft_consensus.cc:383] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.262403 4369 raft_consensus.cc:738] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.262670 4369 consensus_queue.cc:260] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.263473 4369 ts_tablet_manager.cc:1428] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 01:55:09.263762 4369 tablet_bootstrap.cc:492] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.265565 4369 tablet_bootstrap.cc:654] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.269335 4375 tablet_bootstrap.cc:492] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.269562 4375 ts_tablet_manager.cc:1397] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250904 01:55:09.276182 4375 raft_consensus.cc:357] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.276523 4375 raft_consensus.cc:383] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.276654 4375 raft_consensus.cc:738] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.277318 4375 consensus_queue.cc:260] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.278466 4370 ts_tablet_manager.cc:1428] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.038s user 0.010s sys 0.011s
I20250904 01:55:09.278771 4369 tablet_bootstrap.cc:492] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.278946 4369 ts_tablet_manager.cc:1397] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20250904 01:55:09.279670 4369 raft_consensus.cc:357] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.279876 4369 raft_consensus.cc:383] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.280006 4369 raft_consensus.cc:738] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.280107 4370 tablet_bootstrap.cc:492] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.280284 4369 consensus_queue.cc:260] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.280887 4369 ts_tablet_manager.cc:1428] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 01:55:09.282135 4370 tablet_bootstrap.cc:654] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.285415 4301 heartbeater.cc:499] Master 127.3.77.60:38761 was elected leader, sending a full tablet report...
I20250904 01:55:09.286273 4375 ts_tablet_manager.cc:1428] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20250904 01:55:09.287513 4375 tablet_bootstrap.cc:492] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.289026 4375 tablet_bootstrap.cc:654] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.289126 4370 tablet_bootstrap.cc:492] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.289325 4370 ts_tablet_manager.cc:1397] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250904 01:55:09.291021 4375 tablet_bootstrap.cc:492] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.291169 4375 ts_tablet_manager.cc:1397] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250904 01:55:09.292013 4375 raft_consensus.cc:357] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.292199 4375 raft_consensus.cc:383] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.292279 4375 raft_consensus.cc:738] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.292506 4375 consensus_queue.cc:260] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.292788 3784 catalog_manager.cc:2149] created HMS catalog entry for table [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:09.293905 4375 ts_tablet_manager.cc:1428] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 01:55:09.295603 4375 tablet_bootstrap.cc:492] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.297011 4375 tablet_bootstrap.cc:654] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.297691 4370 raft_consensus.cc:357] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.297906 4370 raft_consensus.cc:383] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.297991 4370 raft_consensus.cc:738] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.298355 4370 consensus_queue.cc:260] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.302189 4370 ts_tablet_manager.cc:1428] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250904 01:55:09.305450 4370 tablet_bootstrap.cc:492] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.307425 4370 tablet_bootstrap.cc:654] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.309705 4028 heartbeater.cc:499] Master 127.3.77.60:38761 was elected leader, sending a full tablet report...
I20250904 01:55:09.316685 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 5742132feece4a58a803d0769ee51d05 (DEFAULT_TABLE table=default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.317075 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5742132feece4a58a803d0769ee51d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.319988 3784 master_service.cc:948] ReplaceTablet: received request to replace tablet b30c0c07cbbf43d9aa616c7a025a0a91 from {username='slave'} at 127.0.0.1:43486
I20250904 01:55:09.324930 3962 tablet_service.cc:1468] Processing CreateTablet for tablet 5742132feece4a58a803d0769ee51d05 (DEFAULT_TABLE table=default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.324960 4234 tablet_service.cc:1468] Processing CreateTablet for tablet 5742132feece4a58a803d0769ee51d05 (DEFAULT_TABLE table=default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.325289 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5742132feece4a58a803d0769ee51d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.325284 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5742132feece4a58a803d0769ee51d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.325924 4375 tablet_bootstrap.cc:492] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.326056 4375 ts_tablet_manager.cc:1397] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.009s
I20250904 01:55:09.326833 4375 raft_consensus.cc:357] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.326988 4375 raft_consensus.cc:383] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.327060 4375 raft_consensus.cc:738] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.327262 4375 consensus_queue.cc:260] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.328049 4375 ts_tablet_manager.cc:1428] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 01:55:09.332006 4375 tablet_bootstrap.cc:492] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.332803 4369 tablet_bootstrap.cc:492] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.335007 4369 tablet_bootstrap.cc:654] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.335327 3784 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Sending DeleteTablet for 3 replicas of tablet b30c0c07cbbf43d9aa616c7a025a0a91
I20250904 01:55:09.335915 3784 catalog_manager.cc:6652] ReplaceTablet: tablet b30c0c07cbbf43d9aa616c7a025a0a91 deleted and replaced by tablet 2fc2b696071e46398d928ca25a05488e
I20250904 01:55:09.338706 4370 tablet_bootstrap.cc:492] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.338850 4370 ts_tablet_manager.cc:1397] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.003s
I20250904 01:55:09.339174 4375 tablet_bootstrap.cc:654] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.340288 4370 raft_consensus.cc:357] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.340493 4370 raft_consensus.cc:383] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.340593 4370 raft_consensus.cc:738] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.340870 4370 consensus_queue.cc:260] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.341499 4100 tablet_service.cc:1515] Processing DeleteTablet for tablet b30c0c07cbbf43d9aa616c7a025a0a91 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fc2b696071e46398d928ca25a05488e) from {username='slave'} at 127.0.0.1:35918
I20250904 01:55:09.342710 4370 ts_tablet_manager.cc:1428] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250904 01:55:09.343600 4234 tablet_service.cc:1515] Processing DeleteTablet for tablet b30c0c07cbbf43d9aa616c7a025a0a91 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fc2b696071e46398d928ca25a05488e) from {username='slave'} at 127.0.0.1:36186
I20250904 01:55:09.343592 3962 tablet_service.cc:1515] Processing DeleteTablet for tablet b30c0c07cbbf43d9aa616c7a025a0a91 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fc2b696071e46398d928ca25a05488e) from {username='slave'} at 127.0.0.1:41788
I20250904 01:55:09.343732 4369 tablet_bootstrap.cc:492] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.343936 4369 ts_tablet_manager.cc:1397] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20250904 01:55:09.345026 4369 raft_consensus.cc:357] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.345270 4369 raft_consensus.cc:383] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.345360 4369 raft_consensus.cc:738] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.345821 4369 consensus_queue.cc:260] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.346356 4369 ts_tablet_manager.cc:1428] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 01:55:09.347486 4370 tablet_bootstrap.cc:492] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.349112 4370 tablet_bootstrap.cc:654] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.352283 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 2fc2b696071e46398d928ca25a05488e (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.352672 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fc2b696071e46398d928ca25a05488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.355546 4388 tablet_replica.cc:331] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: stopping tablet replica
I20250904 01:55:09.356174 4388 raft_consensus.cc:2241] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:09.356665 4388 raft_consensus.cc:2270] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:09.358453 4388 ts_tablet_manager.cc:1905] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:09.359005 4235 tablet_service.cc:1468] Processing CreateTablet for tablet 2fc2b696071e46398d928ca25a05488e (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.359377 4235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fc2b696071e46398d928ca25a05488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.362720 3963 tablet_service.cc:1468] Processing CreateTablet for tablet 2fc2b696071e46398d928ca25a05488e (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.363090 3963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fc2b696071e46398d928ca25a05488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.365331 4375 tablet_bootstrap.cc:492] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.365486 4375 ts_tablet_manager.cc:1397] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.011s
W20250904 01:55:09.365537 4169 tablet.cc:2378] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:55:09.366210 4375 raft_consensus.cc:357] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.366457 4375 raft_consensus.cc:383] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.366559 4375 raft_consensus.cc:738] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.366856 4375 consensus_queue.cc:260] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.375188 4369 tablet_bootstrap.cc:492] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.375959 4375 ts_tablet_manager.cc:1428] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 01:55:09.377077 4369 tablet_bootstrap.cc:654] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.377532 4375 tablet_bootstrap.cc:492] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.379693 4375 tablet_bootstrap.cc:654] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.380050 4369 tablet_bootstrap.cc:492] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.380214 4369 ts_tablet_manager.cc:1397] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 01:55:09.381181 4369 raft_consensus.cc:357] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.381414 4369 raft_consensus.cc:383] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.381525 4369 raft_consensus.cc:738] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.381812 4369 consensus_queue.cc:260] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.386859 4369 ts_tablet_manager.cc:1428] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250904 01:55:09.389456 4375 tablet_bootstrap.cc:492] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.389588 4375 ts_tablet_manager.cc:1397] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20250904 01:55:09.390621 4375 raft_consensus.cc:357] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.390786 4375 raft_consensus.cc:383] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.390897 4375 raft_consensus.cc:738] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.391144 4375 consensus_queue.cc:260] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.391978 4388 ts_tablet_manager.cc:1918] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250904 01:55:09.392184 4388 log.cc:1199] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/wals/b30c0c07cbbf43d9aa616c7a025a0a91
I20250904 01:55:09.392695 4388 ts_tablet_manager.cc:1939] T b30c0c07cbbf43d9aa616c7a025a0a91 P 5b23e87ab60248f391fb17326f5e3934: Deleting consensus metadata
I20250904 01:55:09.394119 4375 ts_tablet_manager.cc:1428] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250904 01:55:09.394840 3778 catalog_manager.cc:4928] TS 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911): tablet b30c0c07cbbf43d9aa616c7a025a0a91 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]) successfully deleted
I20250904 01:55:09.399564 3784 master_service.cc:948] ReplaceTablet: received request to replace tablet b30c0c07cbbf43d9aa616c7a025a0a91 from {username='slave'} at 127.0.0.1:43486
I20250904 01:55:09.400570 4389 tablet_replica.cc:331] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: stopping tablet replica
I20250904 01:55:09.400970 4389 raft_consensus.cc:2241] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:09.401296 4389 raft_consensus.cc:2270] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:09.402566 4370 tablet_bootstrap.cc:492] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.402796 4370 ts_tablet_manager.cc:1397] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.025s
I20250904 01:55:09.403595 4370 raft_consensus.cc:357] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.403878 4370 raft_consensus.cc:383] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.404009 4370 raft_consensus.cc:738] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.404297 4370 consensus_queue.cc:260] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.403956 3784 master_service.cc:948] ReplaceTablet: received request to replace tablet b30c0c07cbbf43d9aa616c7a025a0a91 from {username='slave'} at 127.0.0.1:43486
I20250904 01:55:09.404907 4370 ts_tablet_manager.cc:1428] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 01:55:09.405313 4370 tablet_bootstrap.cc:492] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.410498 4389 ts_tablet_manager.cc:1905] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:09.410681 3783 catalog_manager.cc:2149] created HMS catalog entry for table [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:09.413123 4370 tablet_bootstrap.cc:654] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.420037 3783 master_service.cc:948] ReplaceTablet: received request to replace tablet b30c0c07cbbf43d9aa616c7a025a0a91 from {username='slave'} at 127.0.0.1:43486
I20250904 01:55:09.420497 3783 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_5a0ac2833a3d4a2b98f538f7416ca2d1" } new_table_name: "default.Table_ae25e811a20d449d8d8c57cca63f758b" modify_external_catalogs: true
I20250904 01:55:09.422415 3784 master_service.cc:948] ReplaceTablet: received request to replace tablet 735f984a7ed34c758c9032d28e3421ea from {username='slave'} at 127.0.0.1:43486
I20250904 01:55:09.422456 4389 ts_tablet_manager.cc:1918] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250904 01:55:09.422591 4389 log.cc:1199] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/wals/b30c0c07cbbf43d9aa616c7a025a0a91
I20250904 01:55:09.422986 4389 ts_tablet_manager.cc:1939] T b30c0c07cbbf43d9aa616c7a025a0a91 P 7d96d300478849e3b11402fbd34fef15: Deleting consensus metadata
I20250904 01:55:09.423137 4390 tablet_replica.cc:331] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: stopping tablet replica
I20250904 01:55:09.423576 4390 raft_consensus.cc:2241] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:09.423926 4390 raft_consensus.cc:2270] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:09.425493 4390 ts_tablet_manager.cc:1905] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:09.425633 3777 catalog_manager.cc:4928] TS 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819): tablet b30c0c07cbbf43d9aa616c7a025a0a91 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]) successfully deleted
I20250904 01:55:09.426740 4370 tablet_bootstrap.cc:492] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.426885 4370 ts_tablet_manager.cc:1397] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250904 01:55:09.427598 4370 raft_consensus.cc:357] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
W20250904 01:55:09.427698 4031 tablet.cc:2378] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:55:09.427785 4370 raft_consensus.cc:383] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.427882 4370 raft_consensus.cc:738] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.428095 4370 consensus_queue.cc:260] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.430785 3962 tablet_service.cc:1468] Processing CreateTablet for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (DEFAULT_TABLE table=default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.430929 4234 tablet_service.cc:1468] Processing CreateTablet for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (DEFAULT_TABLE table=default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.431253 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd66dd528eb947ceb75bcbd518eb2ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.431288 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd66dd528eb947ceb75bcbd518eb2ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.431345 4100 tablet_service.cc:1468] Processing CreateTablet for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (DEFAULT_TABLE table=default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.431694 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd66dd528eb947ceb75bcbd518eb2ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.437665 4370 ts_tablet_manager.cc:1428] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250904 01:55:09.437960 4369 tablet_bootstrap.cc:492] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.439550 4369 tablet_bootstrap.cc:654] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.440276 4370 tablet_bootstrap.cc:492] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.441815 4370 tablet_bootstrap.cc:654] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.445675 4390 ts_tablet_manager.cc:1918] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250904 01:55:09.445823 3784 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Sending DeleteTablet for 3 replicas of tablet 735f984a7ed34c758c9032d28e3421ea
I20250904 01:55:09.446337 4390 log.cc:1199] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/wals/b30c0c07cbbf43d9aa616c7a025a0a91
I20250904 01:55:09.446857 4235 tablet_service.cc:1515] Processing DeleteTablet for tablet 735f984a7ed34c758c9032d28e3421ea with delete_type TABLET_DATA_DELETED (replaced by tablet 65637e7e27e44ed7ace78227be5e4a50) from {username='slave'} at 127.0.0.1:36186
I20250904 01:55:09.446921 3784 catalog_manager.cc:6652] ReplaceTablet: tablet 735f984a7ed34c758c9032d28e3421ea deleted and replaced by tablet 65637e7e27e44ed7ace78227be5e4a50
I20250904 01:55:09.447209 4369 tablet_bootstrap.cc:492] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.447392 4369 ts_tablet_manager.cc:1397] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20250904 01:55:09.448109 4369 raft_consensus.cc:357] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.448361 4369 raft_consensus.cc:383] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.448472 4369 raft_consensus.cc:738] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.448792 4389 tablet_replica.cc:331] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: stopping tablet replica
I20250904 01:55:09.448734 4369 consensus_queue.cc:260] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.448982 4389 raft_consensus.cc:2241] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:09.449133 4389 raft_consensus.cc:2270] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:09.449545 4100 tablet_service.cc:1515] Processing DeleteTablet for tablet 735f984a7ed34c758c9032d28e3421ea with delete_type TABLET_DATA_DELETED (replaced by tablet 65637e7e27e44ed7ace78227be5e4a50) from {username='slave'} at 127.0.0.1:35918
I20250904 01:55:09.449677 4375 tablet_bootstrap.cc:492] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.449985 4388 tablet_replica.cc:331] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: stopping tablet replica
I20250904 01:55:09.450150 4388 raft_consensus.cc:2241] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:09.450323 4388 raft_consensus.cc:2270] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:09.451447 4369 ts_tablet_manager.cc:1428] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250904 01:55:09.451865 4390 ts_tablet_manager.cc:1939] T b30c0c07cbbf43d9aa616c7a025a0a91 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting consensus metadata
I20250904 01:55:09.452666 3786 catalog_manager.cc:2149] created HMS catalog entry for table [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:09.453379 4388 ts_tablet_manager.cc:1905] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:09.453565 3777 catalog_manager.cc:4928] TS 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169): tablet b30c0c07cbbf43d9aa616c7a025a0a91 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]) successfully deleted
I20250904 01:55:09.454576 3962 tablet_service.cc:1515] Processing DeleteTablet for tablet 735f984a7ed34c758c9032d28e3421ea with delete_type TABLET_DATA_DELETED (replaced by tablet 65637e7e27e44ed7ace78227be5e4a50) from {username='slave'} at 127.0.0.1:41788
I20250904 01:55:09.454775 4390 tablet_replica.cc:331] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: stopping tablet replica
I20250904 01:55:09.454957 4390 raft_consensus.cc:2241] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:09.455091 4390 raft_consensus.cc:2270] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:09.455698 4390 ts_tablet_manager.cc:1905] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:09.458735 4388 ts_tablet_manager.cc:1918] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250904 01:55:09.458847 4388 log.cc:1199] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/wals/735f984a7ed34c758c9032d28e3421ea
I20250904 01:55:09.459144 4388 ts_tablet_manager.cc:1939] T 735f984a7ed34c758c9032d28e3421ea P 5b23e87ab60248f391fb17326f5e3934: Deleting consensus metadata
I20250904 01:55:09.460251 3784 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_782e3736e3434cafa6e2825f14c5900b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:09.460709 4390 ts_tablet_manager.cc:1918] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250904 01:55:09.460824 4390 log.cc:1199] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/wals/735f984a7ed34c758c9032d28e3421ea
I20250904 01:55:09.461158 4390 ts_tablet_manager.cc:1939] T 735f984a7ed34c758c9032d28e3421ea P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting consensus metadata
I20250904 01:55:09.461793 4234 tablet_service.cc:1468] Processing CreateTablet for tablet 65637e7e27e44ed7ace78227be5e4a50 (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.462019 3777 catalog_manager.cc:4928] TS 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169): tablet 735f984a7ed34c758c9032d28e3421ea (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]) successfully deleted
I20250904 01:55:09.462697 3778 catalog_manager.cc:4928] TS 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911): tablet 735f984a7ed34c758c9032d28e3421ea (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]) successfully deleted
I20250904 01:55:09.462807 4370 tablet_bootstrap.cc:492] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.462941 4370 ts_tablet_manager.cc:1397] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.002s
I20250904 01:55:09.463079 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 65637e7e27e44ed7ace78227be5e4a50 (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.463438 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65637e7e27e44ed7ace78227be5e4a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.463387 3962 tablet_service.cc:1468] Processing CreateTablet for tablet 65637e7e27e44ed7ace78227be5e4a50 (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.463725 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65637e7e27e44ed7ace78227be5e4a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.463728 4370 raft_consensus.cc:357] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.463922 4370 raft_consensus.cc:383] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.463996 4370 raft_consensus.cc:738] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.465193 4370 consensus_queue.cc:260] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.467909 4370 ts_tablet_manager.cc:1428] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250904 01:55:09.470598 4370 tablet_bootstrap.cc:492] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.472118 4370 tablet_bootstrap.cc:654] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.475517 3786 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_bc5b38d3266841d6b4a43d4983cff6b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:09.476838 4389 ts_tablet_manager.cc:1905] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:09.477170 4369 tablet_bootstrap.cc:492] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.478798 4375 tablet_bootstrap.cc:654] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.479094 4369 tablet_bootstrap.cc:654] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.479727 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65637e7e27e44ed7ace78227be5e4a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.481453 4370 tablet_bootstrap.cc:492] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.481576 4370 ts_tablet_manager.cc:1397] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20250904 01:55:09.482342 4370 raft_consensus.cc:357] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.482506 4370 raft_consensus.cc:383] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.482582 4370 raft_consensus.cc:738] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.482811 4370 consensus_queue.cc:260] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.485797 4370 ts_tablet_manager.cc:1428] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250904 01:55:09.487679 4389 ts_tablet_manager.cc:1918] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250904 01:55:09.487803 4389 log.cc:1199] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/wals/735f984a7ed34c758c9032d28e3421ea
I20250904 01:55:09.488162 4389 ts_tablet_manager.cc:1939] T 735f984a7ed34c758c9032d28e3421ea P 7d96d300478849e3b11402fbd34fef15: Deleting consensus metadata
I20250904 01:55:09.489133 3777 catalog_manager.cc:4928] TS 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819): tablet 735f984a7ed34c758c9032d28e3421ea (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]) successfully deleted
I20250904 01:55:09.489723 4369 tablet_bootstrap.cc:492] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.489868 4369 ts_tablet_manager.cc:1397] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20250904 01:55:09.490062 4375 tablet_bootstrap.cc:492] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.490191 4375 ts_tablet_manager.cc:1397] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.002s
I20250904 01:55:09.490741 3785 catalog_manager.cc:2149] created HMS catalog entry for table [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:09.490677 4369 raft_consensus.cc:357] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.490840 4369 raft_consensus.cc:383] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.490933 4369 raft_consensus.cc:738] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.494693 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 30378e1466de4f559ac840b676310486 (DEFAULT_TABLE table=default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.494956 3962 tablet_service.cc:1468] Processing CreateTablet for tablet 30378e1466de4f559ac840b676310486 (DEFAULT_TABLE table=default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.495070 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30378e1466de4f559ac840b676310486. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.495441 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30378e1466de4f559ac840b676310486. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.497716 4375 raft_consensus.cc:357] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.497918 4375 raft_consensus.cc:383] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.498003 4375 raft_consensus.cc:738] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.498281 4375 consensus_queue.cc:260] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.498790 4375 ts_tablet_manager.cc:1428] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250904 01:55:09.494999 4369 consensus_queue.cc:260] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.503401 4369 ts_tablet_manager.cc:1428] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250904 01:55:09.505982 4369 tablet_bootstrap.cc:492] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.506584 4370 tablet_bootstrap.cc:492] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.508124 4370 tablet_bootstrap.cc:654] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.512315 4375 tablet_bootstrap.cc:492] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.514092 4375 tablet_bootstrap.cc:654] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.515515 4234 tablet_service.cc:1468] Processing CreateTablet for tablet 30378e1466de4f559ac840b676310486 (DEFAULT_TABLE table=default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.515940 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30378e1466de4f559ac840b676310486. 1 dirs total, 0 dirs full, 0 dirs failed
W20250904 01:55:09.516078 4304 tablet.cc:2378] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:55:09.517503 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 138000c0bebd4169a7d2db4a7f660845 (DEFAULT_TABLE table=default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.517850 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138000c0bebd4169a7d2db4a7f660845. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.518795 3785 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43486:
name: "default.Table_bce07055a7064408b762f1534d59a90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:09.519726 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--webserver_interface=127.3.77.62
--webserver_port=46147
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 01:55:09.521999 4377 raft_consensus.cc:491] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.522210 4377 raft_consensus.cc:513] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.522518 4373 raft_consensus.cc:491] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.522722 4373 raft_consensus.cc:513] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.524024 4377 leader_election.cc:290] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.524598 4373 leader_election.cc:290] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.528034 4235 tablet_service.cc:1468] Processing CreateTablet for tablet 138000c0bebd4169a7d2db4a7f660845 (DEFAULT_TABLE table=default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.528402 4235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138000c0bebd4169a7d2db4a7f660845. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.532111 3962 tablet_service.cc:1468] Processing CreateTablet for tablet 138000c0bebd4169a7d2db4a7f660845 (DEFAULT_TABLE table=default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.532469 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138000c0bebd4169a7d2db4a7f660845. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.536325 4369 tablet_bootstrap.cc:654] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.539865 3789 catalog_manager.cc:2149] created HMS catalog entry for table [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:09.548286 3789 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_18be6a726bc64f8c8efad17c700b66aa" } new_table_name: "default.Table_db0617ec7a994e45b7e7ed190308ddbe" modify_external_catalogs: true
I20250904 01:55:09.559227 4235 tablet_service.cc:1468] Processing CreateTablet for tablet 9f05d628c97e4f57ad4cf4337fe6810f (DEFAULT_TABLE table=default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.560926 4235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f05d628c97e4f57ad4cf4337fe6810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.561087 3963 tablet_service.cc:1468] Processing CreateTablet for tablet 9f05d628c97e4f57ad4cf4337fe6810f (DEFAULT_TABLE table=default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.561455 3963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f05d628c97e4f57ad4cf4337fe6810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.576687 3983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65637e7e27e44ed7ace78227be5e4a50" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" is_pre_election: true
I20250904 01:55:09.577062 3983 raft_consensus.cc:2466] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 0.
I20250904 01:55:09.578747 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 9f05d628c97e4f57ad4cf4337fe6810f (DEFAULT_TABLE table=default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.581513 4375 tablet_bootstrap.cc:492] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.581735 4375 ts_tablet_manager.cc:1397] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.003s
I20250904 01:55:09.583083 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd783e309a44b039890d508fb577e9d" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:09.583181 4375 raft_consensus.cc:357] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.583495 4375 raft_consensus.cc:383] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.583531 4120 raft_consensus.cc:2466] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.583638 4375 raft_consensus.cc:738] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.584018 4375 consensus_queue.cc:260] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.584188 3917 leader_election.cc:304] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.584532 4373 raft_consensus.cc:2802] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.584656 4373 raft_consensus.cc:491] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.584759 4373 raft_consensus.cc:3058] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.584946 3790 catalog_manager.cc:2149] created HMS catalog entry for table [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:09.586609 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65637e7e27e44ed7ace78227be5e4a50" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:09.586995 4256 raft_consensus.cc:2466] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 0.
I20250904 01:55:09.587541 4373 raft_consensus.cc:513] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.588167 4373 leader_election.cc:290] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.588647 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f05d628c97e4f57ad4cf4337fe6810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.590026 4375 ts_tablet_manager.cc:1428] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250904 01:55:09.590919 4375 tablet_bootstrap.cc:492] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.593534 3790 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:09.593858 3790 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:09.598589 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd783e309a44b039890d508fb577e9d" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:09.598790 4120 raft_consensus.cc:3058] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.599217 4375 tablet_bootstrap.cc:654] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.601611 4120 raft_consensus.cc:2466] T cdd783e309a44b039890d508fb577e9d P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.606542 4099 tablet_service.cc:1468] Processing CreateTablet for tablet ba52aec4f85c4723b600ab9271beb725 (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.606832 3963 tablet_service.cc:1468] Processing CreateTablet for tablet ba52aec4f85c4723b600ab9271beb725 (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.606948 4099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba52aec4f85c4723b600ab9271beb725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.607153 3963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba52aec4f85c4723b600ab9271beb725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.608758 4054 leader_election.cc:304] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:09.609120 4377 raft_consensus.cc:2802] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.609230 4377 raft_consensus.cc:491] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.609302 4377 raft_consensus.cc:3058] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.610349 4369 tablet_bootstrap.cc:492] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.610502 4369 ts_tablet_manager.cc:1397] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.000s
I20250904 01:55:09.610567 3917 leader_election.cc:304] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.611323 4373 raft_consensus.cc:2802] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.611564 4369 raft_consensus.cc:357] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.611805 4369 raft_consensus.cc:383] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.611893 4369 raft_consensus.cc:738] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.611963 4377 raft_consensus.cc:513] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.612115 4369 consensus_queue.cc:260] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.612529 4377 leader_election.cc:290] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Requested vote from peers 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.614886 4235 tablet_service.cc:1468] Processing CreateTablet for tablet ba52aec4f85c4723b600ab9271beb725 (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:09.615203 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd783e309a44b039890d508fb577e9d" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:09.615379 4256 raft_consensus.cc:2466] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.615773 4235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba52aec4f85c4723b600ab9271beb725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:09.615862 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd783e309a44b039890d508fb577e9d" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:09.616204 4255 raft_consensus.cc:3058] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.620342 4255 raft_consensus.cc:2466] T cdd783e309a44b039890d508fb577e9d P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.621400 3983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65637e7e27e44ed7ace78227be5e4a50" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
I20250904 01:55:09.621562 3983 raft_consensus.cc:3058] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.625144 3983 raft_consensus.cc:2466] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 1.
I20250904 01:55:09.625494 4054 leader_election.cc:304] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.625553 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65637e7e27e44ed7ace78227be5e4a50" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:09.625757 4377 raft_consensus.cc:2802] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.625916 4255 raft_consensus.cc:3058] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.628723 4255 raft_consensus.cc:2466] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 1.
I20250904 01:55:09.631347 4370 tablet_bootstrap.cc:492] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.631515 4370 ts_tablet_manager.cc:1397] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.125s user 0.022s sys 0.037s
I20250904 01:55:09.632385 4370 raft_consensus.cc:357] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.632640 4370 raft_consensus.cc:383] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.632730 4370 raft_consensus.cc:738] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.632987 4370 consensus_queue.cc:260] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.641281 4369 ts_tablet_manager.cc:1428] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.031s user 0.001s sys 0.004s
I20250904 01:55:09.641654 4369 tablet_bootstrap.cc:492] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.643383 4369 tablet_bootstrap.cc:654] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.643810 4377 raft_consensus.cc:695] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Becoming Leader. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Running, Role: LEADER
I20250904 01:55:09.644275 4377 consensus_queue.cc:237] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.646934 4410 raft_consensus.cc:491] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.647112 4410 raft_consensus.cc:513] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.647482 4373 raft_consensus.cc:695] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:09.648098 4410 leader_election.cc:290] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.648137 4373 consensus_queue.cc:237] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.651320 3983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc2b696071e46398d928ca25a05488e" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" is_pre_election: true
I20250904 01:55:09.651572 3983 raft_consensus.cc:2466] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 0.
I20250904 01:55:09.652336 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc2b696071e46398d928ca25a05488e" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:09.652563 4255 raft_consensus.cc:2466] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 0.
I20250904 01:55:09.653003 4054 leader_election.cc:304] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:09.653316 4410 raft_consensus.cc:2802] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.653482 4410 raft_consensus.cc:491] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.653631 4410 raft_consensus.cc:3058] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.657146 4410 raft_consensus.cc:513] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.659126 4370 ts_tablet_manager.cc:1428] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.028s user 0.005s sys 0.005s
I20250904 01:55:09.659464 4370 tablet_bootstrap.cc:492] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.660571 4410 leader_election.cc:290] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Requested vote from peers 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.661211 3983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc2b696071e46398d928ca25a05488e" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
I20250904 01:55:09.661430 4370 tablet_bootstrap.cc:654] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.661422 3983 raft_consensus.cc:3058] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.664680 3983 raft_consensus.cc:2466] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 1.
I20250904 01:55:09.665745 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc2b696071e46398d928ca25a05488e" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:09.665897 4255 raft_consensus.cc:3058] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.666272 4054 leader_election.cc:304] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.668433 4369 tablet_bootstrap.cc:492] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.668503 4255 raft_consensus.cc:2466] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 1.
I20250904 01:55:09.668574 4369 ts_tablet_manager.cc:1397] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.002s
I20250904 01:55:09.669399 4369 raft_consensus.cc:357] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.669572 4369 raft_consensus.cc:383] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.669703 4369 raft_consensus.cc:738] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.669821 4410 raft_consensus.cc:2802] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.669939 4410 raft_consensus.cc:695] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Becoming Leader. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Running, Role: LEADER
I20250904 01:55:09.670181 4410 consensus_queue.cc:237] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.674324 4375 tablet_bootstrap.cc:492] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.674572 4375 ts_tablet_manager.cc:1397] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.004s
I20250904 01:55:09.675287 4375 raft_consensus.cc:357] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.675506 4375 raft_consensus.cc:383] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.675591 4375 raft_consensus.cc:738] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.675812 4375 consensus_queue.cc:260] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.679486 4375 ts_tablet_manager.cc:1428] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250904 01:55:09.682117 4375 tablet_bootstrap.cc:492] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.683683 4375 tablet_bootstrap.cc:654] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.692947 4375 tablet_bootstrap.cc:492] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.693074 4375 ts_tablet_manager.cc:1397] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250904 01:55:09.694191 4375 raft_consensus.cc:357] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.694416 4375 raft_consensus.cc:383] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.694515 4375 raft_consensus.cc:738] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.694737 4375 consensus_queue.cc:260] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.696004 4369 consensus_queue.cc:260] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.696523 4369 ts_tablet_manager.cc:1428] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250904 01:55:09.696820 4369 tablet_bootstrap.cc:492] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.702780 4381 raft_consensus.cc:491] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.702944 4381 raft_consensus.cc:513] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.704278 4370 tablet_bootstrap.cc:492] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.704455 4370 ts_tablet_manager.cc:1397] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.045s user 0.018s sys 0.001s
I20250904 01:55:09.704859 4381 leader_election.cc:290] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169), 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:09.705227 4370 raft_consensus.cc:357] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.705397 4370 raft_consensus.cc:383] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.705508 4370 raft_consensus.cc:738] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.705716 4370 consensus_queue.cc:260] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.707521 4370 ts_tablet_manager.cc:1428] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 01:55:09.708161 4369 tablet_bootstrap.cc:654] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.708540 4370 tablet_bootstrap.cc:492] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.710163 4370 tablet_bootstrap.cc:654] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.718173 4369 tablet_bootstrap.cc:492] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.718336 4369 ts_tablet_manager.cc:1397] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250904 01:55:09.718976 4369 raft_consensus.cc:357] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.719138 4369 raft_consensus.cc:383] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.719223 4369 raft_consensus.cc:738] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.719398 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5742132feece4a58a803d0769ee51d05" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:09.719436 4369 consensus_queue.cc:260] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.719647 4120 raft_consensus.cc:2466] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 0.
I20250904 01:55:09.719910 4369 ts_tablet_manager.cc:1428] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 01:55:09.720207 4369 tablet_bootstrap.cc:492] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:09.720510 4191 leader_election.cc:304] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:09.721756 4369 tablet_bootstrap.cc:654] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.725106 4381 raft_consensus.cc:2802] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.725224 4381 raft_consensus.cc:491] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.725314 4381 raft_consensus.cc:3058] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.725546 4370 tablet_bootstrap.cc:492] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.725697 4370 ts_tablet_manager.cc:1397] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20250904 01:55:09.726402 4370 raft_consensus.cc:357] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.726635 4370 raft_consensus.cc:383] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.726716 4370 raft_consensus.cc:738] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.726981 4370 consensus_queue.cc:260] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.729427 4370 ts_tablet_manager.cc:1428] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250904 01:55:09.730414 4370 tablet_bootstrap.cc:492] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:09.730648 4381 raft_consensus.cc:513] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.732157 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5742132feece4a58a803d0769ee51d05" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:09.732334 4120 raft_consensus.cc:3058] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.733251 4370 tablet_bootstrap.cc:654] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.735260 4120 raft_consensus.cc:2466] T 5742132feece4a58a803d0769ee51d05 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 1.
I20250904 01:55:09.735687 4191 leader_election.cc:304] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:09.752441 4381 leader_election.cc:290] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 election: Requested vote from peers 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169), 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:09.752768 4381 raft_consensus.cc:2802] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.752972 4381 raft_consensus.cc:695] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [term 1 LEADER]: Becoming Leader. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Running, Role: LEADER
I20250904 01:55:09.753538 4381 consensus_queue.cc:237] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } }
I20250904 01:55:09.762198 4375 ts_tablet_manager.cc:1428] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
I20250904 01:55:09.765568 4375 tablet_bootstrap.cc:492] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.767680 4375 tablet_bootstrap.cc:654] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
W20250904 01:55:09.783609 4191 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.77.2:40911, user_credentials={real_user=slave}} blocked reactor thread for 47910.7us
I20250904 01:55:09.786594 3983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5742132feece4a58a803d0769ee51d05" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" is_pre_election: true
I20250904 01:55:09.786803 3983 raft_consensus.cc:2466] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 0.
I20250904 01:55:09.787555 3983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5742132feece4a58a803d0769ee51d05" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
I20250904 01:55:09.787715 3983 raft_consensus.cc:3058] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.790387 4373 raft_consensus.cc:491] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.790524 4373 raft_consensus.cc:513] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.791062 4373 leader_election.cc:290] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.792106 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd66dd528eb947ceb75bcbd518eb2ee4" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:09.792286 4255 raft_consensus.cc:2466] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.792732 3916 leader_election.cc:304] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:09.793051 4373 raft_consensus.cc:2802] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.793155 4373 raft_consensus.cc:491] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.793272 4373 raft_consensus.cc:3058] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.793615 4408 raft_consensus.cc:491] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.793767 4408 raft_consensus.cc:513] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.794557 4408 leader_election.cc:290] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.795454 4119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138000c0bebd4169a7d2db4a7f660845" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:09.795701 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138000c0bebd4169a7d2db4a7f660845" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:09.795791 4119 raft_consensus.cc:2466] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.795933 4255 raft_consensus.cc:2466] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.796413 3916 leader_election.cc:304] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:09.796717 4408 raft_consensus.cc:2802] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.796838 4408 raft_consensus.cc:491] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.796927 4408 raft_consensus.cc:3058] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.797438 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd66dd528eb947ceb75bcbd518eb2ee4" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:09.797652 4120 raft_consensus.cc:2466] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.798664 3983 raft_consensus.cc:2466] T 5742132feece4a58a803d0769ee51d05 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 1.
I20250904 01:55:09.802402 4373 raft_consensus.cc:513] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.805763 4408 raft_consensus.cc:513] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.806777 4119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138000c0bebd4169a7d2db4a7f660845" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:09.806972 4119 raft_consensus.cc:3058] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.810822 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138000c0bebd4169a7d2db4a7f660845" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:09.810992 4255 raft_consensus.cc:3058] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.814147 4255 raft_consensus.cc:2466] T 138000c0bebd4169a7d2db4a7f660845 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.814488 4408 leader_election.cc:290] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.814824 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd66dd528eb947ceb75bcbd518eb2ee4" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:09.814975 4120 raft_consensus.cc:3058] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.815349 4119 raft_consensus.cc:2466] T 138000c0bebd4169a7d2db4a7f660845 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.815862 3917 leader_election.cc:304] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.819149 4120 raft_consensus.cc:2466] T fd66dd528eb947ceb75bcbd518eb2ee4 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.819685 3917 leader_election.cc:304] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.819981 4409 raft_consensus.cc:2802] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.820577 4408 raft_consensus.cc:2802] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.821552 4373 leader_election.cc:290] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.824805 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd66dd528eb947ceb75bcbd518eb2ee4" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:09.825057 4255 raft_consensus.cc:3058] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.828280 4255 raft_consensus.cc:2466] T fd66dd528eb947ceb75bcbd518eb2ee4 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.839565 4373 raft_consensus.cc:491] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.839803 4373 raft_consensus.cc:513] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.840466 4409 raft_consensus.cc:695] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:09.840798 4409 consensus_queue.cc:237] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.841228 4373 leader_election.cc:290] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.841503 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30378e1466de4f559ac840b676310486" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:09.841725 4120 raft_consensus.cc:2466] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.842190 3917 leader_election.cc:304] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.842619 4373 raft_consensus.cc:2802] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.842760 4373 raft_consensus.cc:491] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.842909 4373 raft_consensus.cc:3058] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.846278 4373 raft_consensus.cc:513] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.847513 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30378e1466de4f559ac840b676310486" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:09.847772 4120 raft_consensus.cc:3058] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.850355 4120 raft_consensus.cc:2466] T 30378e1466de4f559ac840b676310486 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.850766 3917 leader_election.cc:304] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.851150 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30378e1466de4f559ac840b676310486" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:09.851384 4255 raft_consensus.cc:2466] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.851468 4428 raft_consensus.cc:2802] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.851595 4428 raft_consensus.cc:695] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:09.851912 4428 consensus_queue.cc:237] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.856362 4375 tablet_bootstrap.cc:492] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.856585 4375 ts_tablet_manager.cc:1397] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.004s
I20250904 01:55:09.857528 4375 raft_consensus.cc:357] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.858438 4373 leader_election.cc:290] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
W20250904 01:55:09.859552 3776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.77.62:34273, user_credentials={real_user=slave}} blocked reactor thread for 75317.7us
I20250904 01:55:09.861004 4408 raft_consensus.cc:695] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:09.861443 4408 consensus_queue.cc:237] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:09.858714 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30378e1466de4f559ac840b676310486" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:09.862715 4255 raft_consensus.cc:3058] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.865990 4255 raft_consensus.cc:2466] T 30378e1466de4f559ac840b676310486 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.866693 4375 raft_consensus.cc:383] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.866896 4375 raft_consensus.cc:738] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.868184 4369 tablet_bootstrap.cc:492] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:09.868417 4369 ts_tablet_manager.cc:1397] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.148s user 0.038s sys 0.095s
I20250904 01:55:09.869642 4369 raft_consensus.cc:357] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.870543 4375 consensus_queue.cc:260] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.870769 4369 raft_consensus.cc:383] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.870857 4370 tablet_bootstrap.cc:492] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:09.870896 4369 raft_consensus.cc:738] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.870994 4370 ts_tablet_manager.cc:1397] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.141s user 0.023s sys 0.036s
I20250904 01:55:09.871937 4370 raft_consensus.cc:357] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.872100 4370 raft_consensus.cc:383] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.872179 4370 raft_consensus.cc:738] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.872438 4370 consensus_queue.cc:260] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.874012 4375 ts_tablet_manager.cc:1428] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.017s user 0.000s sys 0.001s
I20250904 01:55:09.874274 4369 consensus_queue.cc:260] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.874963 4369 ts_tablet_manager.cc:1428] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250904 01:55:09.876575 4370 ts_tablet_manager.cc:1428] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250904 01:55:09.877171 4375 tablet_bootstrap.cc:492] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:09.878983 4375 tablet_bootstrap.cc:654] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:09.889698 4375 tablet_bootstrap.cc:492] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:09.889843 4375 ts_tablet_manager.cc:1397] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20250904 01:55:09.890663 4375 raft_consensus.cc:357] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.890844 4375 raft_consensus.cc:383] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:09.890952 4375 raft_consensus.cc:738] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:09.891180 4375 consensus_queue.cc:260] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.893711 4375 ts_tablet_manager.cc:1428] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
01:55:09.913 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_ac1709b4bdce4074b1fac1b503f0d518 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250904 01:55:09.915688 3787 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=05306578f265474c9f716a8ecc1531d9]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_ac1709b4bdce4074b1fac1b503f0d518 already exists)
I20250904 01:55:09.916079 3787 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:43486 (request call id 12) took 1073 ms. Trace:
I20250904 01:55:09.916198 3787 rpcz_store.cc:276] 0904 01:55:08.842883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0904 01:55:08.843451 (+ 568us) service_pool.cc:225] Handling call
0904 01:55:09.088388 (+244937us) catalog_manager.cc:2076] Acquired catalog manager lock
0904 01:55:09.088800 (+ 412us) catalog_manager.cc:2122] Created new table and tablet info
0904 01:55:09.916046 (+827246us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":495917,"Hive Metastore.run_cpu_time_us":709,"Hive Metastore.run_wall_time_us":330651}
I20250904 01:55:09.918907 3787 catalog_manager.cc:5582] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2). New cstate: current_term: 1 leader_uuid: "5b23e87ab60248f391fb17326f5e3934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } health_report { overall_health: UNKNOWN } } }
I20250904 01:55:09.926185 3787 catalog_manager.cc:5582] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1). New cstate: current_term: 1 leader_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } health_report { overall_health: UNKNOWN } } }
I20250904 01:55:09.932891 3787 catalog_manager.cc:5582] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d96d300478849e3b11402fbd34fef15 (127.3.77.3). New cstate: current_term: 1 leader_uuid: "7d96d300478849e3b11402fbd34fef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } health_report { overall_health: UNKNOWN } } }
F20250904 01:55:09.936036 3883 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7efd716fadcd google::LogMessage::Fail() at ??:0
@ 0x7efd716feb93 google::LogMessage::SendToLog() at ??:0
@ 0x7efd716fa7cc google::LogMessage::Flush() at ??:0
I20250904 01:55:09.944149 3787 catalog_manager.cc:5582] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2). New cstate: current_term: 1 leader_uuid: "5b23e87ab60248f391fb17326f5e3934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } health_report { overall_health: UNKNOWN } } }
@ 0x7efd716faa11 google::LogMessage::~LogMessage() at ??:0
I20250904 01:55:09.951941 3787 catalog_manager.cc:5582] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1). New cstate: current_term: 1 leader_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } health_report { overall_health: UNKNOWN } } }
I20250904 01:55:09.952616 3787 catalog_manager.cc:5582] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1). New cstate: current_term: 1 leader_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } health_report { overall_health: UNKNOWN } } }
@ 0x7efd87b3d7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 01:55:09.953289 3787 catalog_manager.cc:5582] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1). New cstate: current_term: 1 leader_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } health_report { overall_health: UNKNOWN } } }
@ 0x7efd87b3c061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250904 01:55:09.959554 3787 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:09.959753 3787 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43486:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
@ 0x7efd87b43cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7efd87b43b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7efd744632b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efd778876db start_thread at ??:0
@ 0x7efd705ab71f clone at ??:0
I20250904 01:55:09.980187 4408 raft_consensus.cc:491] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:09.980387 4408 raft_consensus.cc:513] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.981027 4408 leader_election.cc:290] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.981534 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba52aec4f85c4723b600ab9271beb725" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:09.981756 4120 raft_consensus.cc:2466] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.981890 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba52aec4f85c4723b600ab9271beb725" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:09.982115 4255 raft_consensus.cc:2466] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:09.982257 3917 leader_election.cc:304] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.982715 4408 raft_consensus.cc:2802] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:09.982781 4408 raft_consensus.cc:491] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:09.982843 4408 raft_consensus.cc:3058] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.984555 4408 raft_consensus.cc:513] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:09.985338 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba52aec4f85c4723b600ab9271beb725" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:09.985347 4255 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba52aec4f85c4723b600ab9271beb725" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:09.985496 4120 raft_consensus.cc:3058] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.985496 4255 raft_consensus.cc:3058] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:09.988189 4255 raft_consensus.cc:2466] T ba52aec4f85c4723b600ab9271beb725 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.988191 4120 raft_consensus.cc:2466] T ba52aec4f85c4723b600ab9271beb725 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:09.988621 4408 leader_election.cc:290] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:09.988583 3917 leader_election.cc:304] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:09.988912 4408 raft_consensus.cc:2802] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:09.990032 4408 raft_consensus.cc:695] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:09.990343 4408 consensus_queue.cc:237] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
W20250904 01:55:10.201745 4403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:10.202080 4403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:10.202258 4403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:10.210161 4403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:10.210255 4403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:10.210341 4403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:10.210405 4403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:10.219432 4403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.77.62
--webserver_port=46147
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:10.220948 4403 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20250904 01:55:10.220942 3915 connection.cc:537] client connection to 127.3.77.60:38761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:10.221359 4308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.60:38761)
W20250904 01:55:10.222059 4308 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.60:38761)
W20250904 01:55:10.222620 4308 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.60:38761)
I20250904 01:55:10.223001 4403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:10.223282 4308 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.60:38761)
W20250904 01:55:10.231431 4454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:10.231448 4453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:10.232237 4456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:10.233727 4403 server_base.cc:1047] running on GCE node
I20250904 01:55:10.234529 4403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:10.235713 4403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:10.236891 4403 hybrid_clock.cc:648] HybridClock initialized: now 1756950910236847 us; error 41 us; skew 500 ppm
I20250904 01:55:10.237156 4403 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:10.239034 4403 webserver.cc:480] Webserver started at http://127.3.77.62:46147/ using document root <none> and password file <none>
I20250904 01:55:10.239637 4403 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:10.239737 4403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:10.244864 4403 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20250904 01:55:10.247328 4462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:10.248487 4403 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250904 01:55:10.248651 4403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:10.249213 4403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:10.269101 4403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:10.269742 4403 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:10.270001 4403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:10.288851 4403 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.62:34273
I20250904 01:55:10.288867 4513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.62:34273 every 8 connection(s)
I20250904 01:55:10.290284 4403 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
I20250904 01:55:10.294727 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 4403
I20250904 01:55:10.296290 4514 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:10.299600 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap starting.
I20250904 01:55:10.308471 4514 log.cc:826] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:10.316025 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:55:10.316416 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap complete.
I20250904 01:55:10.320789 4514 raft_consensus.cc:357] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:10.321282 4514 raft_consensus.cc:738] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Initialized, Role: FOLLOWER
I20250904 01:55:10.321754 4514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:10.322335 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:10.322554 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:10.323057 4514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:10.343513 4533 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:10.343622 4533 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: loading cluster ID for follower catalog manager: success
I20250904 01:55:10.350446 4533 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: acquiring CA information for follower catalog manager: success
I20250904 01:55:10.352697 4533 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:10.366110 4403 master_runner.cc:186] Error getting master registration for 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111),
I20250904 01:55:10.369242 4403 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:10.383412 4167 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:10.383584 4167 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
I20250904 01:55:10.386438 4478 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:41287; Asking this server to re-register.
I20250904 01:55:10.387074 4167 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:10.388348 4478 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:10.400789 4302 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:10.400929 4302 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
I20250904 01:55:10.402384 4478 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:43983; Asking this server to re-register.
I20250904 01:55:10.402925 4302 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:10.404049 4478 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:10.424228 4408 consensus_queue.cc:1035] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.433879 4408 consensus_queue.cc:1035] T cdd783e309a44b039890d508fb577e9d P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.440183 4029 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:10.442808 4478 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:53465; Asking this server to re-register.
I20250904 01:55:10.444010 4029 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:10.445533 4478 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:10.459995 4548 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f298ba82a5e49028407388810de4b71)
I20250904 01:55:10.460157 4548 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:10.460811 4548 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:10.464993 3706 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:10.468014 4489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
I20250904 01:55:10.468495 4489 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef804bcecd943d89c7af93e9375b1f3 in term 1.
I20250904 01:55:10.468961 3706 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:10.469195 4548 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 01:55:10.469307 4548 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7f298ba82a5e49028407388810de4b71)
I20250904 01:55:10.469404 4548 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:55:10.471468 4548 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:10.472113 4548 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 3 election: Requested vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:10.472710 4489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc"
I20250904 01:55:10.472913 4489 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 1 FOLLOWER]: Advancing to term 3
W20250904 01:55:10.473656 3706 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:10.476899 4489 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef804bcecd943d89c7af93e9375b1f3 in term 3.
I20250904 01:55:10.477222 3706 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:10.477485 4548 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 3 FOLLOWER]: Leader election won for term 3
I20250904 01:55:10.477928 4548 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 3 LEADER]: Becoming Leader. State: Replica: 6ef804bcecd943d89c7af93e9375b1f3, State: Running, Role: LEADER
I20250904 01:55:10.478411 4548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:10.479672 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef804bcecd943d89c7af93e9375b1f3. Latest consensus state: current_term: 3 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:10.479913 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:10.480587 4553 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:10.482882 4553 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:10.483309 4553 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:10.483681 4553 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:10.484118 4553 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:10.484503 4553 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:10.484860 4553 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:10.485172 4553 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:10.485505 4553 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:10.485814 4553 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:10.487386 4553 tablet_loader.cc:96] loaded metadata for tablet 138000c0bebd4169a7d2db4a7f660845 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:10.487735 4553 tablet_loader.cc:96] loaded metadata for tablet 2fc2b696071e46398d928ca25a05488e (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:10.488050 4553 tablet_loader.cc:96] loaded metadata for tablet 30378e1466de4f559ac840b676310486 (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:10.488324 4553 tablet_loader.cc:96] loaded metadata for tablet 5742132feece4a58a803d0769ee51d05 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:10.488616 4553 tablet_loader.cc:96] loaded metadata for tablet 65637e7e27e44ed7ace78227be5e4a50 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:10.489054 4553 tablet_loader.cc:96] loaded metadata for tablet 9f05d628c97e4f57ad4cf4337fe6810f (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:10.489495 4553 tablet_loader.cc:96] loaded metadata for tablet ba52aec4f85c4723b600ab9271beb725 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:10.489774 4553 tablet_loader.cc:96] loaded metadata for tablet cdd783e309a44b039890d508fb577e9d (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:10.490056 4553 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:10.490307 4553 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:10.491173 4553 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:10.491240 4553 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:10.492399 4553 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:10.493268 4553 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Loaded TSK: 0
I20250904 01:55:10.494397 4553 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:10.495694 4553 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:10.496559 4553 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:10.522665 3719 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51488:
name: "default.Table_968a17a9054a48dda6494188adb73474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:10.523192 3714 master_service.cc:948] ReplaceTablet: received request to replace tablet cdd783e309a44b039890d508fb577e9d from {username='slave'} at 127.0.0.1:51488
I20250904 01:55:10.523236 3718 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51488:
name: "default.Table_c245a71ea8e74145959886fcfac66f07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:10.523669 3717 master_service.cc:948] ReplaceTablet: received request to replace tablet cdd783e309a44b039890d508fb577e9d from {username='slave'} at 127.0.0.1:51488
I20250904 01:55:10.523756 3722 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51488:
name: "default.Table_338af7b0a5ed48918a49777a432224f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:10.523874 3716 master_service.cc:948] ReplaceTablet: received request to replace tablet ba52aec4f85c4723b600ab9271beb725 from {username='slave'} at 127.0.0.1:51488
I20250904 01:55:10.524099 3715 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51488:
name: "default.Table_d03c97312adf4523be9539809e8fdb5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:10.525035 3713 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51488:
table { table_name: "default.Table_18be6a726bc64f8c8efad17c700b66aa" } new_table_name: "default.Table_db0617ec7a994e45b7e7ed190308ddbe" modify_external_catalogs: true
I20250904 01:55:10.522711 3720 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51488:
name: "default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:10.525825 3721 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51488:
name: "default.Table_d723b3eb51ff4772a7331c9338cb1918"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 01:55:10.529965 3706 consensus_peers.cc:489] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 -> Peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Couldn't send request to peer 7f298ba82a5e49028407388810de4b71. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:55:10.531370 4489 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Refusing update from remote peer 6ef804bcecd943d89c7af93e9375b1f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250904 01:55:10.532227 4548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 4, Time since last communication: 0.000s
F20250904 01:55:10.532536 3848 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fae3eb04dcd google::LogMessage::Fail() at ??:0
@ 0x7fae3eb08b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fae3eb047cc google::LogMessage::Flush() at ??:0
@ 0x7fae3eb04a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fae54f477b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fae54f46061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fae54f4dcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fae54f4db59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fae4186d2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fae44c916db start_thread at ??:0
@ 0x7fae3d9b571f clone at ??:0
I20250904 01:55:10.574963 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef804bcecd943d89c7af93e9375b1f3. Latest consensus state: current_term: 3 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:10.575275 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:10.575726 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:10.576036 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:10.600230 4556 mvcc.cc:204] Tried to move back new op lower bound from 7196470927175294976 to 7196470922490241024. Current Snapshot: MvccSnapshot[applied={T|T < 7196470927175294976 or (T in {7196470927175294976})}]
I20250904 01:55:10.634997 4429 consensus_queue.cc:1035] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.641920 4408 consensus_queue.cc:1035] T 138000c0bebd4169a7d2db4a7f660845 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.656109 4545 consensus_queue.cc:1035] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.666661 4575 consensus_queue.cc:1035] T 5742132feece4a58a803d0769ee51d05 P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.671118 4540 consensus_queue.cc:1035] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.677572 4540 consensus_queue.cc:1035] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.699534 4540 consensus_queue.cc:1035] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.700345 4540 consensus_queue.cc:1035] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.715286 4408 raft_consensus.cc:491] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:10.715459 4408 raft_consensus.cc:513] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:10.716092 4408 leader_election.cc:290] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819), 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:10.716656 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f05d628c97e4f57ad4cf4337fe6810f" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:10.716812 4256 raft_consensus.cc:2466] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:10.717229 3916 leader_election.cc:304] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:10.717537 4408 raft_consensus.cc:2802] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:10.717613 4408 raft_consensus.cc:491] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:10.717698 4408 raft_consensus.cc:3058] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:10.720367 4408 raft_consensus.cc:513] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:10.721385 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f05d628c97e4f57ad4cf4337fe6810f" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:10.721539 4256 raft_consensus.cc:3058] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:10.723210 4408 leader_election.cc:290] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819), 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:10.724442 4582 raft_consensus.cc:491] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:10.724643 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f05d628c97e4f57ad4cf4337fe6810f" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:10.724642 4582 raft_consensus.cc:513] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:10.725988 4582 leader_election.cc:290] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:10.726286 4120 raft_consensus.cc:2466] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:10.726935 4256 raft_consensus.cc:2466] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:10.727361 3916 leader_election.cc:304] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:10.728061 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f05d628c97e4f57ad4cf4337fe6810f" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:10.728410 4256 raft_consensus.cc:2391] T 9f05d628c97e4f57ad4cf4337fe6810f P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b23e87ab60248f391fb17326f5e3934 in current term 1: Already voted for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in this term.
I20250904 01:55:10.729552 3981 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f05d628c97e4f57ad4cf4337fe6810f" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" is_pre_election: true
I20250904 01:55:10.729929 3981 raft_consensus.cc:2391] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b23e87ab60248f391fb17326f5e3934 in current term 1: Already voted for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in this term.
I20250904 01:55:10.730712 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f05d628c97e4f57ad4cf4337fe6810f" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:10.730909 4120 raft_consensus.cc:3058] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:10.734877 4408 raft_consensus.cc:2802] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:10.734201 4120 raft_consensus.cc:2466] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:10.735713 4408 raft_consensus.cc:695] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:10.734772 4054 leader_election.cc:304] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934; no voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15
I20250904 01:55:10.736418 4582 raft_consensus.cc:2747] T 9f05d628c97e4f57ad4cf4337fe6810f P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 01:55:10.737334 4408 consensus_queue.cc:237] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
W20250904 01:55:10.785203 4054 connection.cc:537] client connection to 127.3.77.61:36307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:10.785398 4190 connection.cc:537] client connection to 127.3.77.61:36307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:10.788659 3916 connection.cc:537] client connection to 127.3.77.61:36307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:10.789510 4463 connection.cc:537] server connection from 127.0.0.1:47052 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:10.789994 4539 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:10.786128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.61:36307 (local address 127.3.77.3:41841)
0904 01:55:10.786392 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:10.786408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:10.786519 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:10.789147 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":149}
W20250904 01:55:10.790644 4538 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:10.786128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.61:36307 (local address 127.3.77.2:54911)
0904 01:55:10.786387 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:10.786408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:10.786519 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:10.790016 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":149}
W20250904 01:55:10.791200 4450 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:10.787961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.61:36307 (local address 127.0.0.1:51514)
0904 01:55:10.788159 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:10.788175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:10.788274 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:10.790959 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":99}
I20250904 01:55:10.844734 4408 consensus_queue.cc:1035] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:10.849319 4408 consensus_queue.cc:1035] T 30378e1466de4f559ac840b676310486 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:11.009617 4429 consensus_queue.cc:1035] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:11.014671 4544 consensus_queue.cc:1035] T fd66dd528eb947ceb75bcbd518eb2ee4 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:11.038268 4560 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:11.038465 4560 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:11.040072 4560 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:11.040791 4463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:11.041913 4463 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:11.042100 4464 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:11.042250 4464 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:11.042548 4560 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:55:11.093911 4544 consensus_queue.cc:1035] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:11.098817 4544 consensus_queue.cc:1035] T ba52aec4f85c4723b600ab9271beb725 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:11.663064 4544 consensus_queue.cc:1035] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:11.667754 4544 consensus_queue.cc:1035] T 9f05d628c97e4f57ad4cf4337fe6810f P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:11.754379 4613 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:11.754531 4613 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:11.755115 4613 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:11.756691 4464 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:11.756819 4463 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:11.756968 4463 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:11.757190 4613 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:55:12.633461 4623 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:12.633633 4623 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:12.634119 4623 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:12.635840 4464 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:12.635959 4463 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:12.636085 4463 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:12.636281 4623 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:55:13.330797 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.60:38761
--webserver_interface=127.3.77.60
--webserver_port=41119
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:13.618947 4635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:13.619207 4635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:13.619329 4635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:13.626268 4635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:13.626346 4635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:13.626430 4635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:13.626487 4635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:13.635946 4635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.60:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.77.60
--webserver_port=41119
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:13.637231 4635 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:13.638748 4635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:13.646168 4646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:13.646459 4649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:13.647135 4647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:13.647471 4635 server_base.cc:1047] running on GCE node
I20250904 01:55:13.648109 4635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:13.649371 4635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:13.650597 4635 hybrid_clock.cc:648] HybridClock initialized: now 1756950913650511 us; error 62 us; skew 500 ppm
I20250904 01:55:13.650924 4635 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:13.652747 4635 webserver.cc:480] Webserver started at http://127.3.77.60:41119/ using document root <none> and password file <none>
I20250904 01:55:13.653450 4635 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:13.653573 4635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:13.658511 4635 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:55:13.660809 4655 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:13.661898 4635 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250904 01:55:13.662030 4635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
uuid: "7f298ba82a5e49028407388810de4b71"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:13.662642 4635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:13.681605 4635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:13.682785 4635 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:13.683239 4635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:13.710808 4635 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.60:38761
I20250904 01:55:13.710846 4708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.60:38761 every 8 connection(s)
I20250904 01:55:13.712201 4635 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-2/data/info.pb
I20250904 01:55:13.718080 4709 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:13.718209 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 4635
I20250904 01:55:13.721735 4709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Bootstrap starting.
I20250904 01:55:13.731869 4709 log.cc:826] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:13.757236 4709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=23 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:55:13.757683 4709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Bootstrap complete.
I20250904 01:55:13.761952 4709 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:13.762609 4709 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f298ba82a5e49028407388810de4b71, State: Initialized, Role: FOLLOWER
I20250904 01:55:13.763105 4709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:13.763769 4713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:13.763976 4713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:13.764633 4709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:13.776597 4727 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:13.776727 4727 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: loading cluster ID for follower catalog manager: success
I20250904 01:55:13.780774 4727 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: acquiring CA information for follower catalog manager: success
I20250904 01:55:13.783226 4727 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:13.804008 4635 master_runner.cc:186] Error getting master registration for 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111),
I20250904 01:55:13.810401 4635 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:13.879446 4166 heartbeater.cc:344] Connected to a master server at 127.3.77.60:38761
I20250904 01:55:13.882649 4674 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:49637; Asking this server to re-register.
I20250904 01:55:13.883411 4166 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:13.884949 4674 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:14.133157 4736 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:14.133370 4736 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:14.134071 4736 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:14.135907 4464 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:14.137019 4028 heartbeater.cc:344] Connected to a master server at 127.3.77.60:38761
I20250904 01:55:14.138772 4674 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:51027; Asking this server to re-register.
I20250904 01:55:14.139890 4028 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:14.141384 4673 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:14.142447 4684 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "7f298ba82a5e49028407388810de4b71" is_pre_election: true
I20250904 01:55:14.142841 4684 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 2.
I20250904 01:55:14.143246 4463 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71; no voters: 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:14.143471 4736 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 01:55:14.143573 4736 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:14.143684 4736 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 3 FOLLOWER]: Advancing to term 4
I20250904 01:55:14.145442 4736 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:14.145895 4736 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 election: Requested vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:14.146349 4684 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "7f298ba82a5e49028407388810de4b71"
I20250904 01:55:14.146494 4684 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 2 FOLLOWER]: Advancing to term 4
W20250904 01:55:14.147045 4464 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:14.149899 4684 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 4.
I20250904 01:55:14.150285 4463 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71; no voters: 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:14.150575 4736 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 4 FOLLOWER]: Leader election won for term 4
I20250904 01:55:14.150919 4736 raft_consensus.cc:695] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 4 LEADER]: Becoming Leader. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Running, Role: LEADER
I20250904 01:55:14.151288 4736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.35, Last appended by leader: 35, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:14.152621 4741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 4 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:14.152889 4741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:14.197502 4301 heartbeater.cc:344] Connected to a master server at 127.3.77.60:38761
I20250904 01:55:14.198904 4673 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:37405; Asking this server to re-register.
I20250904 01:55:14.199415 4301 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:14.200488 4673 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:14.664988 4745 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:14.665139 4745 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:14.666762 4745 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
I20250904 01:55:14.667135 4489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 5 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
W20250904 01:55:14.668635 4657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:14.669817 4657 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:14.669952 4657 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f298ba82a5e49028407388810de4b71; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:14.670261 4745 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250904 01:55:14.688371 4464 consensus_peers.cc:489] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc -> Peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Couldn't send request to peer 6ef804bcecd943d89c7af93e9375b1f3. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:55:14.703475 4684 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Refusing update from remote peer 29b8a62cc8d04df781f991da2bd83ccc: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20250904 01:55:14.704128 4748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 32, Time since last communication: 0.000s
I20250904 01:55:14.708761 4745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 4 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:14.708899 4745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:14.714763 4760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:14.714851 4745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:14.715077 4760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:14.715116 4745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:14.715906 4748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:14.716104 4748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:14.715991 4754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:14.716229 4754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:14.797566 4743 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:14.800016 4743 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:14.800462 4743 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:14.800817 4743 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:14.801162 4743 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:14.801497 4743 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:14.801827 4743 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:14.802132 4743 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:14.802489 4743 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:14.802806 4743 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:14.804294 4743 tablet_loader.cc:96] loaded metadata for tablet 138000c0bebd4169a7d2db4a7f660845 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:14.804687 4743 tablet_loader.cc:96] loaded metadata for tablet 2fc2b696071e46398d928ca25a05488e (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:14.805018 4743 tablet_loader.cc:96] loaded metadata for tablet 30378e1466de4f559ac840b676310486 (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:14.805266 4743 tablet_loader.cc:96] loaded metadata for tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:14.805543 4743 tablet_loader.cc:96] loaded metadata for tablet 5742132feece4a58a803d0769ee51d05 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:14.805819 4743 tablet_loader.cc:96] loaded metadata for tablet 65637e7e27e44ed7ace78227be5e4a50 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:14.806281 4743 tablet_loader.cc:96] loaded metadata for tablet 9f05d628c97e4f57ad4cf4337fe6810f (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:14.806978 4743 tablet_loader.cc:96] loaded metadata for tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:14.807245 4743 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:14.807453 4743 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:14.808205 4743 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:14.808274 4743 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:14.809279 4743 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:14.810057 4743 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Loaded TSK: 0
I20250904 01:55:14.810838 4743 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:14.811609 4743 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:14.812389 4743 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:14.830734 4479 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b" } modify_external_catalogs: true
I20250904 01:55:14.830967 4479 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b" } modify_external_catalogs: true
I20250904 01:55:14.830991 4478 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47038:
name: "default.Table_39d410e70d9349abad85c4dccbad54bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:14.832015 4473 master_service.cc:948] ReplaceTablet: received request to replace tablet 2fc2b696071e46398d928ca25a05488e from {username='slave'} at 127.0.0.1:47038
I20250904 01:55:14.832644 4471 master_service.cc:948] ReplaceTablet: received request to replace tablet cdd783e309a44b039890d508fb577e9d from {username='slave'} at 127.0.0.1:47038
I20250904 01:55:14.831881 4477 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:14.833772 4477 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:14.833205 4476 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:14.834466 4471 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47038:
name: "default.Table_968a17a9054a48dda6494188adb73474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:14.832631 4475 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47038:
name: "default.Table_bce07055a7064408b762f1534d59a90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:14.835085 4476 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:14.834982 4474 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_5a0ac2833a3d4a2b98f538f7416ca2d1" } new_table_name: "default.Table_ae25e811a20d449d8d8c57cca63f758b" modify_external_catalogs: true
I20250904 01:55:14.834414 4472 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "default.Table_a524c1ec64554d8eb7687bfa84e4579c" } new_table_name: "default.Table_3a626a942a5e41e9b5d523f0be9b7387" modify_external_catalogs: true
I20250904 01:55:14.835018 4470 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47038:
name: "default.Table_c245a71ea8e74145959886fcfac66f07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:14.842674 4473 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Sending DeleteTablet for 3 replicas of tablet 2fc2b696071e46398d928ca25a05488e
F20250904 01:55:14.843008 4532 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f85d041cdcd google::LogMessage::Fail() at ??:0
@ 0x7f85d0420b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f85d041c7cc google::LogMessage::Flush() at ??:0
@ 0x7f85d041ca11 google::LogMessage::~LogMessage() at ??:0
I20250904 01:55:14.855188 4473 catalog_manager.cc:6652] ReplaceTablet: tablet 2fc2b696071e46398d928ca25a05488e deleted and replaced by tablet 4f9d1156f12b43839c00ac99ffdb118e
I20250904 01:55:14.856012 4473 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47038:
name: "default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:14.864905 4235 tablet_service.cc:1515] Processing DeleteTablet for tablet 2fc2b696071e46398d928ca25a05488e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f9d1156f12b43839c00ac99ffdb118e) from {username='slave'} at 127.0.0.1:59166
I20250904 01:55:14.866796 3963 tablet_service.cc:1515] Processing DeleteTablet for tablet 2fc2b696071e46398d928ca25a05488e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f9d1156f12b43839c00ac99ffdb118e) from {username='slave'} at 127.0.0.1:55220
I20250904 01:55:14.867385 4771 tablet_replica.cc:331] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: stopping tablet replica
I20250904 01:55:14.867568 4771 raft_consensus.cc:2241] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:14.867725 4771 raft_consensus.cc:2270] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:14.868263 4771 ts_tablet_manager.cc:1905] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:14.868384 4770 tablet_replica.cc:331] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: stopping tablet replica
I20250904 01:55:14.868570 4770 raft_consensus.cc:2241] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:14.868551 4100 tablet_service.cc:1515] Processing DeleteTablet for tablet 2fc2b696071e46398d928ca25a05488e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f9d1156f12b43839c00ac99ffdb118e) from {username='slave'} at 127.0.0.1:52466
I20250904 01:55:14.868768 4770 raft_consensus.cc:2270] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:14.869119 4772 tablet_replica.cc:331] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: stopping tablet replica
I20250904 01:55:14.869335 4772 raft_consensus.cc:2241] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Raft consensus shutting down.
I20250904 01:55:14.869460 4770 ts_tablet_manager.cc:1905] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:14.869838 4772 raft_consensus.cc:2270] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:14.870460 4772 ts_tablet_manager.cc:1905] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:14.874106 4771 ts_tablet_manager.cc:1918] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:14.874240 4771 log.cc:1199] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/wals/2fc2b696071e46398d928ca25a05488e
I20250904 01:55:14.874702 4771 ts_tablet_manager.cc:1939] T 2fc2b696071e46398d928ca25a05488e P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting consensus metadata
@ 0x7f85e685f7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 01:55:14.875717 4464 catalog_manager.cc:4928] TS 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169): tablet 2fc2b696071e46398d928ca25a05488e (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]) successfully deleted
I20250904 01:55:14.876225 4770 ts_tablet_manager.cc:1918] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:14.876322 4772 ts_tablet_manager.cc:1918] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:14.876344 4770 log.cc:1199] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/wals/2fc2b696071e46398d928ca25a05488e
I20250904 01:55:14.876616 4772 log.cc:1199] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/wals/2fc2b696071e46398d928ca25a05488e
I20250904 01:55:14.876714 4770 ts_tablet_manager.cc:1939] T 2fc2b696071e46398d928ca25a05488e P 7d96d300478849e3b11402fbd34fef15: Deleting consensus metadata
I20250904 01:55:14.877035 4772 ts_tablet_manager.cc:1939] T 2fc2b696071e46398d928ca25a05488e P 5b23e87ab60248f391fb17326f5e3934: Deleting consensus metadata
I20250904 01:55:14.877727 4464 catalog_manager.cc:4928] TS 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819): tablet 2fc2b696071e46398d928ca25a05488e (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]) successfully deleted
I20250904 01:55:14.878017 4465 catalog_manager.cc:4928] TS 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911): tablet 2fc2b696071e46398d928ca25a05488e (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]) successfully deleted
@ 0x7f85e685e061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250904 01:55:14.883047 4100 tablet_service.cc:1468] Processing CreateTablet for tablet 4f9d1156f12b43839c00ac99ffdb118e (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.883033 4235 tablet_service.cc:1468] Processing CreateTablet for tablet 4f9d1156f12b43839c00ac99ffdb118e (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.883340 4100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f9d1156f12b43839c00ac99ffdb118e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:14.883461 4235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f9d1156f12b43839c00ac99ffdb118e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f85e6865cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f85e6865b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250904 01:55:14.891332 3963 tablet_service.cc:1468] Processing CreateTablet for tablet 4f9d1156f12b43839c00ac99ffdb118e (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.891559 4099 tablet_service.cc:1468] Processing CreateTablet for tablet 3fa7e488232a450dbe28c7d2252a7a1c (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.891728 3963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f9d1156f12b43839c00ac99ffdb118e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:14.891930 4099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa7e488232a450dbe28c7d2252a7a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:14.892118 4234 tablet_service.cc:1468] Processing CreateTablet for tablet 3fa7e488232a450dbe28c7d2252a7a1c (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.892431 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa7e488232a450dbe28c7d2252a7a1c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f85d31852b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f85d65a96db start_thread at ??:0
I20250904 01:55:14.899713 3962 tablet_service.cc:1468] Processing CreateTablet for tablet 3fa7e488232a450dbe28c7d2252a7a1c (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.900028 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa7e488232a450dbe28c7d2252a7a1c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f85cf2cd71f clone at ??:0
I20250904 01:55:14.901480 4236 tablet_service.cc:1468] Processing CreateTablet for tablet ce814e705d2a4a5cafb5670424c008f0 (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.901786 4236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce814e705d2a4a5cafb5670424c008f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:14.901906 4098 tablet_service.cc:1468] Processing CreateTablet for tablet ce814e705d2a4a5cafb5670424c008f0 (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:14.902181 4098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce814e705d2a4a5cafb5670424c008f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:14.907071 4777 tablet_bootstrap.cc:492] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:14.908671 4774 tablet_bootstrap.cc:492] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:14.910418 4774 tablet_bootstrap.cc:654] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.911044 4777 tablet_bootstrap.cc:654] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.914732 4777 tablet_bootstrap.cc:492] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:14.914857 4777 ts_tablet_manager.cc:1397] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250904 01:55:14.915807 4774 tablet_bootstrap.cc:492] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:14.915932 4774 ts_tablet_manager.cc:1397] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250904 01:55:14.916615 4777 raft_consensus.cc:357] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.916786 4777 raft_consensus.cc:383] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.916719 4774 raft_consensus.cc:357] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.916858 4777 raft_consensus.cc:738] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.916893 4774 raft_consensus.cc:383] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.916962 4774 raft_consensus.cc:738] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.917054 4777 consensus_queue.cc:260] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.917176 4774 consensus_queue.cc:260] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.920603 4777 ts_tablet_manager.cc:1428] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250904 01:55:14.921154 4774 ts_tablet_manager.cc:1428] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250904 01:55:14.923008 4777 tablet_bootstrap.cc:492] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:14.923530 4774 tablet_bootstrap.cc:492] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:14.925024 4776 tablet_bootstrap.cc:492] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:14.925331 4774 tablet_bootstrap.cc:654] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.927317 4774 tablet_bootstrap.cc:492] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:14.927443 4774 ts_tablet_manager.cc:1397] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 01:55:14.928025 4777 tablet_bootstrap.cc:654] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.928084 4774 raft_consensus.cc:357] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.928272 4774 raft_consensus.cc:383] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.929708 4776 tablet_bootstrap.cc:654] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.928367 4774 raft_consensus.cc:738] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.930696 4774 consensus_queue.cc:260] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.931882 4776 tablet_bootstrap.cc:492] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:14.932003 4776 ts_tablet_manager.cc:1397] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250904 01:55:14.932160 4774 ts_tablet_manager.cc:1428] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250904 01:55:14.932695 4776 raft_consensus.cc:357] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.932883 4776 raft_consensus.cc:383] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.932971 4776 raft_consensus.cc:738] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.933182 4776 consensus_queue.cc:260] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.934759 4776 ts_tablet_manager.cc:1428] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 01:55:14.935631 4774 tablet_bootstrap.cc:492] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:14.936148 4776 tablet_bootstrap.cc:492] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:14.937526 4774 tablet_bootstrap.cc:654] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.937695 4776 tablet_bootstrap.cc:654] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.943145 4777 tablet_bootstrap.cc:492] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:14.943296 4777 ts_tablet_manager.cc:1397] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250904 01:55:14.944176 4777 raft_consensus.cc:357] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.944393 4777 raft_consensus.cc:383] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.944479 4777 raft_consensus.cc:738] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.944797 4777 consensus_queue.cc:260] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.945402 4777 ts_tablet_manager.cc:1428] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 01:55:14.945977 4777 tablet_bootstrap.cc:492] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:14.948761 4777 tablet_bootstrap.cc:654] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:14.951725 4776 tablet_bootstrap.cc:492] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:14.951876 4776 ts_tablet_manager.cc:1397] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250904 01:55:14.952659 4776 raft_consensus.cc:357] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.952864 4776 raft_consensus.cc:383] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.952950 4776 raft_consensus.cc:738] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.953183 4776 consensus_queue.cc:260] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.953624 4776 ts_tablet_manager.cc:1428] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250904 01:55:14.955459 4777 tablet_bootstrap.cc:492] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:14.955637 4777 ts_tablet_manager.cc:1397] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250904 01:55:14.956429 4777 raft_consensus.cc:357] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.956593 4777 raft_consensus.cc:383] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.956677 4777 raft_consensus.cc:738] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.956924 4777 consensus_queue.cc:260] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.957406 4777 ts_tablet_manager.cc:1428] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 01:55:14.962312 4429 raft_consensus.cc:491] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:14.962455 4429 raft_consensus.cc:513] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.963120 4429 leader_election.cc:290] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:14.963758 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f9d1156f12b43839c00ac99ffdb118e" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:14.963989 4256 raft_consensus.cc:2466] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:14.964648 3916 leader_election.cc:304] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:14.964972 4429 raft_consensus.cc:2802] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:14.965130 4429 raft_consensus.cc:491] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:14.965255 4429 raft_consensus.cc:3058] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:14.965225 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f9d1156f12b43839c00ac99ffdb118e" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:14.965423 4120 raft_consensus.cc:2466] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:14.965608 4774 tablet_bootstrap.cc:492] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:14.965747 4774 ts_tablet_manager.cc:1397] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.005s
I20250904 01:55:14.966609 4774 raft_consensus.cc:357] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.966799 4774 raft_consensus.cc:383] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:14.966881 4774 raft_consensus.cc:738] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:14.967203 4774 consensus_queue.cc:260] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.969234 4774 ts_tablet_manager.cc:1428] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 01:55:14.970485 4429 raft_consensus.cc:513] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:14.971117 4429 leader_election.cc:290] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:14.972049 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f9d1156f12b43839c00ac99ffdb118e" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:14.972105 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f9d1156f12b43839c00ac99ffdb118e" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:14.972261 4256 raft_consensus.cc:3058] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:14.972252 4120 raft_consensus.cc:3058] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:14.974942 4256 raft_consensus.cc:2466] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:14.974956 4120 raft_consensus.cc:2466] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:14.975512 3917 leader_election.cc:304] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:14.975937 4429 raft_consensus.cc:2802] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:14.976328 4429 raft_consensus.cc:695] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:14.976644 4429 consensus_queue.cc:237] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
W20250904 01:55:15.095957 4191 connection.cc:537] server connection from 127.0.0.1:59166 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:15.095999 4055 connection.cc:537] server connection from 127.0.0.1:52466 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:15.096228 4189 connection.cc:537] client connection to 127.3.77.62:34273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:15.096634 3915 connection.cc:537] client connection to 127.3.77.62:34273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:15.097415 4308 connection.cc:537] client connection to 127.3.77.62:34273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:15.097575 3917 connection.cc:537] server connection from 127.0.0.1:55220 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:15.098084 4308 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.62:34273)
W20250904 01:55:15.098381 4308 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:15.098547 4308 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:15.098860 4308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.62:34273)
I20250904 01:55:15.123481 4749 raft_consensus.cc:491] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:15.123670 4749 raft_consensus.cc:513] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:15.124300 4749 leader_election.cc:290] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:15.124809 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa7e488232a450dbe28c7d2252a7a1c" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:15.124850 3982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa7e488232a450dbe28c7d2252a7a1c" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" is_pre_election: true
I20250904 01:55:15.125031 4256 raft_consensus.cc:2466] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 0.
I20250904 01:55:15.125057 3982 raft_consensus.cc:2466] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 0.
I20250904 01:55:15.125484 4054 leader_election.cc:304] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:15.125746 4749 raft_consensus.cc:2802] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:15.125867 4749 raft_consensus.cc:491] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:15.125959 4749 raft_consensus.cc:3058] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:15.128258 4749 raft_consensus.cc:513] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:15.128773 4749 leader_election.cc:290] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Requested vote from peers 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:15.129124 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa7e488232a450dbe28c7d2252a7a1c" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:15.129231 3982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa7e488232a450dbe28c7d2252a7a1c" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
I20250904 01:55:15.129287 4256 raft_consensus.cc:3058] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:15.129391 3982 raft_consensus.cc:3058] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:15.131734 4256 raft_consensus.cc:2466] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 1.
I20250904 01:55:15.132047 4054 leader_election.cc:304] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:15.132190 3982 raft_consensus.cc:2466] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 1.
I20250904 01:55:15.132309 4749 raft_consensus.cc:2802] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:15.132736 4749 raft_consensus.cc:695] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Becoming Leader. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Running, Role: LEADER
I20250904 01:55:15.133229 4749 consensus_queue.cc:237] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:15.374691 4749 raft_consensus.cc:491] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:15.374876 4749 raft_consensus.cc:513] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:15.375396 4749 leader_election.cc:290] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:15.375828 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce814e705d2a4a5cafb5670424c008f0" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:15.376031 4256 raft_consensus.cc:2466] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 0.
I20250904 01:55:15.376421 4054 leader_election.cc:304] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:15.376688 4749 raft_consensus.cc:2802] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:15.376770 4749 raft_consensus.cc:491] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:15.376876 4749 raft_consensus.cc:3058] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
W20250904 01:55:15.376801 4054 leader_election.cc:343] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169): Not found: Tablet not found: ce814e705d2a4a5cafb5670424c008f0
I20250904 01:55:15.378685 4749 raft_consensus.cc:513] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:15.379127 4749 leader_election.cc:290] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Requested vote from peers 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:15.379477 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce814e705d2a4a5cafb5670424c008f0" candidate_uuid: "5b23e87ab60248f391fb17326f5e3934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:15.379647 4256 raft_consensus.cc:3058] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
W20250904 01:55:15.379848 4054 leader_election.cc:343] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169): Not found: Tablet not found: ce814e705d2a4a5cafb5670424c008f0
I20250904 01:55:15.381338 4256 raft_consensus.cc:2466] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b23e87ab60248f391fb17326f5e3934 in term 1.
I20250904 01:55:15.381644 4054 leader_election.cc:304] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters: 2ca66c5fca0c4305a2857a0074a8f8cf
I20250904 01:55:15.381887 4749 raft_consensus.cc:2802] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:15.382043 4749 raft_consensus.cc:695] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Becoming Leader. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Running, Role: LEADER
I20250904 01:55:15.382193 4799 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:15.382350 4749 consensus_queue.cc:237] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:15.382580 4799 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:15.383293 4799 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
W20250904 01:55:15.387439 4656 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:15.387567 4657 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:15.387691 4657 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f298ba82a5e49028407388810de4b71; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:15.388468 4799 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250904 01:55:15.909185 4429 consensus_queue.cc:1035] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:15.915016 4811 consensus_queue.cc:1035] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:15.951467 4806 consensus_queue.cc:1035] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:15.956635 4806 consensus_queue.cc:1035] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:16.122398 4821 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:16.122581 4821 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:16.123113 4821 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
W20250904 01:55:16.124917 4656 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:16.125165 4657 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:16.125275 4657 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f298ba82a5e49028407388810de4b71; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:16.125524 4821 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250904 01:55:16.218521 4806 consensus_queue.cc:1035] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 01:55:16.223090 4054 consensus_peers.cc:489] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 -> Peer 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169): Couldn't send request to peer 2ca66c5fca0c4305a2857a0074a8f8cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce814e705d2a4a5cafb5670424c008f0. This is attempt 1: this message will repeat every 5th retry.
I20250904 01:55:16.720832 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--webserver_interface=127.3.77.62
--webserver_port=46147
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:17.016763 4831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:17.017019 4831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:17.017097 4831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:17.024448 4831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:17.024520 4831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:17.024575 4831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:17.024619 4831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:17.033689 4831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.77.62
--webserver_port=46147
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:17.035082 4831 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:17.036599 4831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:17.043917 4840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:17.043955 4841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:17.044802 4843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:17.045261 4831 server_base.cc:1047] running on GCE node
I20250904 01:55:17.045852 4831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:17.047045 4831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:17.048240 4831 hybrid_clock.cc:648] HybridClock initialized: now 1756950917048199 us; error 32 us; skew 500 ppm
I20250904 01:55:17.048558 4831 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:17.050689 4831 webserver.cc:480] Webserver started at http://127.3.77.62:46147/ using document root <none> and password file <none>
I20250904 01:55:17.051328 4831 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:17.051422 4831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:17.056026 4831 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250904 01:55:17.058495 4849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:17.059649 4831 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250904 01:55:17.059785 4831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:17.060348 4831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:17.073882 4831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:17.074610 4831 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:17.074888 4831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:17.092039 4831 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.62:34273
I20250904 01:55:17.092075 4900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.62:34273 every 8 connection(s)
I20250904 01:55:17.093343 4831 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
I20250904 01:55:17.097366 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 4831
I20250904 01:55:17.099144 4901 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:17.102125 4901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap starting.
I20250904 01:55:17.111229 4901 log.cc:826] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:17.136426 4901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=26 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:55:17.136958 4901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap complete.
I20250904 01:55:17.142907 4901 raft_consensus.cc:357] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:17.143445 4901 raft_consensus.cc:738] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.143923 4901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:17.144572 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:17.144793 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:17.145360 4901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:17.157840 4919 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:17.157915 4919 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: loading cluster ID for follower catalog manager: success
I20250904 01:55:17.161674 4919 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: acquiring CA information for follower catalog manager: success
I20250904 01:55:17.164078 4919 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:17.164912 4831 master_runner.cc:186] Error getting master registration for 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111),
I20250904 01:55:17.169791 4831 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:17.180593 4167 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:17.183475 4866 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:59185; Asking this server to re-register.
I20250904 01:55:17.184291 4167 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:17.184998 4925 ts_tablet_manager.cc:927] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Initiating tablet copy from peer 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:17.185962 4866 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:17.189497 4925 tablet_copy_client.cc:323] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet copy: Beginning tablet copy session from remote peer at address 127.3.77.2:40911
I20250904 01:55:17.190402 4141 tablet_copy_service.cc:140] P 5b23e87ab60248f391fb17326f5e3934: Received BeginTabletCopySession request for tablet ce814e705d2a4a5cafb5670424c008f0 from peer 2ca66c5fca0c4305a2857a0074a8f8cf ({username='slave'} at 127.3.77.1:38519)
I20250904 01:55:17.190560 4141 tablet_copy_service.cc:161] P 5b23e87ab60248f391fb17326f5e3934: Beginning new tablet copy session on tablet ce814e705d2a4a5cafb5670424c008f0 from peer 2ca66c5fca0c4305a2857a0074a8f8cf at {username='slave'} at 127.3.77.1:38519: session id = 2ca66c5fca0c4305a2857a0074a8f8cf-ce814e705d2a4a5cafb5670424c008f0
I20250904 01:55:17.192636 4141 tablet_copy_source_session.cc:215] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Tablet Copy: opened 0 blocks and 1 log segments
I20250904 01:55:17.194319 4925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce814e705d2a4a5cafb5670424c008f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:17.200420 4925 tablet_copy_client.cc:806] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet copy: Starting download of 0 data blocks...
I20250904 01:55:17.200766 4925 tablet_copy_client.cc:670] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet copy: Starting download of 1 WAL segments...
I20250904 01:55:17.202641 4925 tablet_copy_client.cc:538] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250904 01:55:17.206153 4925 tablet_bootstrap.cc:492] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:17.212847 4925 tablet_bootstrap.cc:492] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:55:17.213289 4925 tablet_bootstrap.cc:492] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap complete.
I20250904 01:55:17.213526 4925 ts_tablet_manager.cc:1397] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.007s user 0.008s sys 0.000s
I20250904 01:55:17.214416 4925 raft_consensus.cc:357] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.214610 4925 raft_consensus.cc:738] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.214856 4925 consensus_queue.cc:260] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.215410 4925 ts_tablet_manager.cc:1428] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250904 01:55:17.216516 4141 tablet_copy_service.cc:342] P 5b23e87ab60248f391fb17326f5e3934: Request end of tablet copy session 2ca66c5fca0c4305a2857a0074a8f8cf-ce814e705d2a4a5cafb5670424c008f0 received from {username='slave'} at 127.3.77.1:38519
I20250904 01:55:17.216696 4141 tablet_copy_service.cc:434] P 5b23e87ab60248f391fb17326f5e3934: ending tablet copy session 2ca66c5fca0c4305a2857a0074a8f8cf-ce814e705d2a4a5cafb5670424c008f0 on tablet ce814e705d2a4a5cafb5670424c008f0 with peer 2ca66c5fca0c4305a2857a0074a8f8cf
I20250904 01:55:17.221940 4928 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:17.222110 4928 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:17.224668 4029 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:17.226577 4866 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:46129; Asking this server to re-register.
I20250904 01:55:17.227567 4029 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:17.229138 4866 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:17.230505 4928 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
I20250904 01:55:17.240748 4302 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:17.243502 4866 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:34593; Asking this server to re-register.
I20250904 01:55:17.243475 4876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 5 candidate_status { last_received { term: 4 index: 38 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
I20250904 01:55:17.244218 4876 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f298ba82a5e49028407388810de4b71 in term 4.
I20250904 01:55:17.244328 4302 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:17.245133 4656 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71; no voters:
I20250904 01:55:17.245484 4928 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250904 01:55:17.245641 4928 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:17.245756 4928 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 4 FOLLOWER]: Advancing to term 5
I20250904 01:55:17.245862 4866 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:17.249663 4928 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:17.250648 4928 leader_election.cc:290] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 election: Requested vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307)
I20250904 01:55:17.250954 4876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f298ba82a5e49028407388810de4b71" candidate_term: 5 candidate_status { last_received { term: 4 index: 38 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc"
I20250904 01:55:17.251127 4876 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 4 FOLLOWER]: Advancing to term 5
W20250904 01:55:17.255066 4657 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:17.256022 4657 leader_election.cc:336] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:17.256523 4876 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f298ba82a5e49028407388810de4b71 in term 5.
I20250904 01:55:17.256883 4656 leader_election.cc:304] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71; no voters: 6ef804bcecd943d89c7af93e9375b1f3
I20250904 01:55:17.257170 4928 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 5 FOLLOWER]: Leader election won for term 5
I20250904 01:55:17.257561 4928 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [term 5 LEADER]: Becoming Leader. State: Replica: 7f298ba82a5e49028407388810de4b71, State: Running, Role: LEADER
I20250904 01:55:17.258070 4928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:17.259886 4936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f298ba82a5e49028407388810de4b71. Latest consensus state: current_term: 5 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:17.260156 4936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:17.260766 4938 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:17.263163 4938 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:17.263548 4938 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:17.263886 4938 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:17.264221 4938 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:17.264482 4938 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:17.264811 4938 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:17.265134 4938 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:17.265432 4938 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:17.265756 4938 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:17.267330 4938 tablet_loader.cc:96] loaded metadata for tablet 138000c0bebd4169a7d2db4a7f660845 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:17.267882 4938 tablet_loader.cc:96] loaded metadata for tablet 30378e1466de4f559ac840b676310486 (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:17.268152 4938 tablet_loader.cc:96] loaded metadata for tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:17.268409 4938 tablet_loader.cc:96] loaded metadata for tablet 4f9d1156f12b43839c00ac99ffdb118e (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:17.268676 4938 tablet_loader.cc:96] loaded metadata for tablet 5742132feece4a58a803d0769ee51d05 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:17.268934 4938 tablet_loader.cc:96] loaded metadata for tablet 65637e7e27e44ed7ace78227be5e4a50 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:17.269398 4938 tablet_loader.cc:96] loaded metadata for tablet 9f05d628c97e4f57ad4cf4337fe6810f (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:17.270098 4938 tablet_loader.cc:96] loaded metadata for tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:17.270467 4938 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:17.270670 4938 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:17.271466 4938 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:17.271553 4938 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:17.272792 4938 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:17.274019 4938 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Loaded TSK: 0
I20250904 01:55:17.274840 4938 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:17.275698 4938 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:17.276466 4938 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:17.343657 4674 master_service.cc:948] ReplaceTablet: received request to replace tablet ba52aec4f85c4723b600ab9271beb725 from {username='slave'} at 127.0.0.1:54528
I20250904 01:55:17.343700 4673 master_service.cc:948] ReplaceTablet: received request to replace tablet 4f9d1156f12b43839c00ac99ffdb118e from {username='slave'} at 127.0.0.1:54528
I20250904 01:55:17.345853 4674 master_service.cc:948] ReplaceTablet: received request to replace tablet 65637e7e27e44ed7ace78227be5e4a50 from {username='slave'} at 127.0.0.1:54528
I20250904 01:55:17.344650 4671 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_e916036e5adf47c886c74e8088132031" } new_table_name: "default.Table_386e7bb279394e4ba14a48582a9ba8f8" modify_external_catalogs: true
I20250904 01:55:17.344449 4672 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_7d5858a140ad4b7eb83927265abad9fe" } modify_external_catalogs: true
I20250904 01:55:17.346426 4670 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_9a196bab16bb4604ab9b00d45335f022" } new_table_name: "default.Table_09793f14275347b0aaa65279c0b3bc41" modify_external_catalogs: true
I20250904 01:55:17.346954 4672 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_7d5858a140ad4b7eb83927265abad9fe" } modify_external_catalogs: true
I20250904 01:55:17.346789 4669 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54528:
name: "default.Table_782e3736e3434cafa6e2825f14c5900b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:17.348991 4665 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_5a0ac2833a3d4a2b98f538f7416ca2d1" } new_table_name: "default.Table_ae25e811a20d449d8d8c57cca63f758b" modify_external_catalogs: true
I20250904 01:55:17.349167 4667 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_18be6a726bc64f8c8efad17c700b66aa" } new_table_name: "default.Table_db0617ec7a994e45b7e7ed190308ddbe" modify_external_catalogs: true
I20250904 01:55:17.348745 4668 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54528:
name: "default.Table_bc5b38d3266841d6b4a43d4983cff6b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:17.352756 4666 master_service.cc:948] ReplaceTablet: received request to replace tablet cdd783e309a44b039890d508fb577e9d from {username='slave'} at 127.0.0.1:54528
F20250904 01:55:17.353156 4726 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250904 01:55:17.353428 4666 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54528:
name: "default.Table_bce07055a7064408b762f1534d59a90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f943b054dcd google::LogMessage::Fail() at ??:0
@ 0x7f943b058b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f943b0547cc google::LogMessage::Flush() at ??:0
@ 0x7f943b054a11 google::LogMessage::~LogMessage() at ??:0
W20250904 01:55:17.365945 4657 consensus_peers.cc:489] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 -> Peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Couldn't send request to peer 6ef804bcecd943d89c7af93e9375b1f3. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:55:17.367389 4876 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Refusing update from remote peer 7f298ba82a5e49028407388810de4b71: Log matching property violated. Preceding OpId in replica: term: 4 index: 38. Preceding OpId from leader: term: 5 index: 40. (index mismatch)
I20250904 01:55:17.368432 4936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
@ 0x7f94514977b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 01:55:17.375185 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f298ba82a5e49028407388810de4b71. Latest consensus state: current_term: 5 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:17.375403 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9451496061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250904 01:55:17.381651 4942 mvcc.cc:204] Tried to move back new op lower bound from 7196470957518757888 to 7196470957093085184. Current Snapshot: MvccSnapshot[applied={T|T < 7196470957518757888}]
@ 0x7f945149dcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f945149db59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250904 01:55:17.391232 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:17.391438 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:17.395692 4674 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Sending DeleteTablet for 3 replicas of tablet 65637e7e27e44ed7ace78227be5e4a50
I20250904 01:55:17.396739 4936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "7f298ba82a5e49028407388810de4b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:17.396940 4936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:17.397557 4674 catalog_manager.cc:6652] ReplaceTablet: tablet 65637e7e27e44ed7ace78227be5e4a50 deleted and replaced by tablet 04b299dd322a4c35bbed9ec90e9fa3b6
I20250904 01:55:17.397882 4674 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:17.398090 4674 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:17.399389 4673 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7f298ba82a5e49028407388810de4b71: Sending DeleteTablet for 3 replicas of tablet 4f9d1156f12b43839c00ac99ffdb118e
I20250904 01:55:17.399745 4673 catalog_manager.cc:6652] ReplaceTablet: tablet 4f9d1156f12b43839c00ac99ffdb118e deleted and replaced by tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96
I20250904 01:55:17.399994 4673 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:17.400141 4673 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54528:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
@ 0x7f943ddbd2b9 kudu::Thread::SuperviseThread() at ??:0
I20250904 01:55:17.425009 3963 tablet_service.cc:1515] Processing DeleteTablet for tablet 65637e7e27e44ed7ace78227be5e4a50 with delete_type TABLET_DATA_DELETED (replaced by tablet 04b299dd322a4c35bbed9ec90e9fa3b6) from {username='slave'} at 127.0.0.1:55230
I20250904 01:55:17.426831 3962 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f9d1156f12b43839c00ac99ffdb118e with delete_type TABLET_DATA_DELETED (replaced by tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96) from {username='slave'} at 127.0.0.1:55230
I20250904 01:55:17.427505 3962 tablet_service.cc:1468] Processing CreateTablet for tablet 04b299dd322a4c35bbed9ec90e9fa3b6 (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f94411e16db start_thread at ??:0
I20250904 01:55:17.427914 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b299dd322a4c35bbed9ec90e9fa3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:17.428270 4235 tablet_service.cc:1515] Processing DeleteTablet for tablet 65637e7e27e44ed7ace78227be5e4a50 with delete_type TABLET_DATA_DELETED (replaced by tablet 04b299dd322a4c35bbed9ec90e9fa3b6) from {username='slave'} at 127.0.0.1:59176
I20250904 01:55:17.428570 4236 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f9d1156f12b43839c00ac99ffdb118e with delete_type TABLET_DATA_DELETED (replaced by tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96) from {username='slave'} at 127.0.0.1:59176
I20250904 01:55:17.428894 4234 tablet_service.cc:1468] Processing CreateTablet for tablet 04b299dd322a4c35bbed9ec90e9fa3b6 (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f9439f0571f clone at ??:0
I20250904 01:55:17.429204 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b299dd322a4c35bbed9ec90e9fa3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:17.431030 4098 tablet_service.cc:1515] Processing DeleteTablet for tablet 65637e7e27e44ed7ace78227be5e4a50 with delete_type TABLET_DATA_DELETED (replaced by tablet 04b299dd322a4c35bbed9ec90e9fa3b6) from {username='slave'} at 127.0.0.1:52476
I20250904 01:55:17.431589 4097 tablet_service.cc:1468] Processing CreateTablet for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:17.431891 4097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:17.433827 4958 tablet_bootstrap.cc:492] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:17.434051 3961 tablet_service.cc:1468] Processing CreateTablet for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:17.434461 3961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:17.434432 4100 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f9d1156f12b43839c00ac99ffdb118e with delete_type TABLET_DATA_DELETED (replaced by tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96) from {username='slave'} at 127.0.0.1:52476
I20250904 01:55:17.434968 4236 tablet_service.cc:1468] Processing CreateTablet for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (DEFAULT_TABLE table=default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:17.435312 4236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:17.435796 4958 tablet_bootstrap.cc:654] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:17.436652 4099 tablet_service.cc:1468] Processing CreateTablet for tablet 04b299dd322a4c35bbed9ec90e9fa3b6 (DEFAULT_TABLE table=default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:17.437031 4099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b299dd322a4c35bbed9ec90e9fa3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:17.444294 4958 tablet_bootstrap.cc:492] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:17.444443 4958 ts_tablet_manager.cc:1397] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250904 01:55:17.445344 4958 raft_consensus.cc:357] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.445513 4958 raft_consensus.cc:383] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:17.445608 4958 raft_consensus.cc:738] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.445801 4958 consensus_queue.cc:260] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.447194 4958 ts_tablet_manager.cc:1428] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250904 01:55:17.447487 4958 tablet_bootstrap.cc:492] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:17.457862 4958 tablet_bootstrap.cc:654] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:17.460903 4961 tablet_bootstrap.cc:492] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:17.462752 4961 tablet_bootstrap.cc:654] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:17.463300 4956 tablet_replica.cc:331] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: stopping tablet replica
I20250904 01:55:17.463541 4956 raft_consensus.cc:2241] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:17.463804 4956 raft_consensus.cc:2270] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:17.464716 4956 ts_tablet_manager.cc:1905] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:17.467887 4955 tablet_replica.cc:331] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: stopping tablet replica
I20250904 01:55:17.468070 4955 raft_consensus.cc:2241] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:17.468298 4955 raft_consensus.cc:2270] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:17.468731 4955 ts_tablet_manager.cc:1905] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:17.469332 4958 tablet_bootstrap.cc:492] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:17.469434 4958 ts_tablet_manager.cc:1397] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20250904 01:55:17.469910 4958 raft_consensus.cc:357] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.470033 4958 raft_consensus.cc:383] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:17.470083 4958 raft_consensus.cc:738] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.470351 4958 consensus_queue.cc:260] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.471094 4959 tablet_bootstrap.cc:492] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:17.472834 4959 tablet_bootstrap.cc:654] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:17.472965 4958 ts_tablet_manager.cc:1428] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 01:55:17.474731 4961 tablet_bootstrap.cc:492] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:17.474865 4961 ts_tablet_manager.cc:1397] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250904 01:55:17.475641 4961 raft_consensus.cc:357] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.475821 4961 raft_consensus.cc:383] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:17.475821 4955 ts_tablet_manager.cc:1918] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:17.475912 4955 log.cc:1199] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/wals/65637e7e27e44ed7ace78227be5e4a50
I20250904 01:55:17.475914 4961 raft_consensus.cc:738] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.476119 4961 consensus_queue.cc:260] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.476219 4955 ts_tablet_manager.cc:1939] T 65637e7e27e44ed7ace78227be5e4a50 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting consensus metadata
I20250904 01:55:17.477092 4955 tablet_replica.cc:331] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: stopping tablet replica
I20250904 01:55:17.478945 4961 ts_tablet_manager.cc:1428] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250904 01:55:17.479216 4961 tablet_bootstrap.cc:492] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:17.480940 4961 tablet_bootstrap.cc:654] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:17.477273 4955 raft_consensus.cc:2241] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Raft consensus shutting down.
I20250904 01:55:17.481403 4955 raft_consensus.cc:2270] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:17.481863 4955 ts_tablet_manager.cc:1905] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:17.482240 4956 ts_tablet_manager.cc:1918] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:17.482343 4956 log.cc:1199] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/wals/65637e7e27e44ed7ace78227be5e4a50
I20250904 01:55:17.482553 4959 tablet_bootstrap.cc:492] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:17.482681 4959 ts_tablet_manager.cc:1397] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250904 01:55:17.482748 4956 ts_tablet_manager.cc:1939] T 65637e7e27e44ed7ace78227be5e4a50 P 7d96d300478849e3b11402fbd34fef15: Deleting consensus metadata
I20250904 01:55:17.483548 4956 tablet_replica.cc:331] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: stopping tablet replica
I20250904 01:55:17.483431 4959 raft_consensus.cc:357] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.483633 4959 raft_consensus.cc:383] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:17.483716 4959 raft_consensus.cc:738] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.483727 4956 raft_consensus.cc:2241] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:17.483867 4956 raft_consensus.cc:2270] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:17.484310 4956 ts_tablet_manager.cc:1905] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:17.487203 4961 tablet_bootstrap.cc:492] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:17.487341 4961 ts_tablet_manager.cc:1397] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20250904 01:55:17.487645 4955 ts_tablet_manager.cc:1918] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:17.487764 4955 log.cc:1199] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/wals/4f9d1156f12b43839c00ac99ffdb118e
I20250904 01:55:17.487915 4961 raft_consensus.cc:357] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.488075 4961 raft_consensus.cc:383] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:17.488164 4961 raft_consensus.cc:738] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.488168 4955 ts_tablet_manager.cc:1939] T 4f9d1156f12b43839c00ac99ffdb118e P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting consensus metadata
I20250904 01:55:17.488416 4957 tablet_replica.cc:331] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: stopping tablet replica
I20250904 01:55:17.488368 4961 consensus_queue.cc:260] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.488581 4957 raft_consensus.cc:2241] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Raft consensus shutting down.
I20250904 01:55:17.488909 4957 raft_consensus.cc:2270] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:17.489399 4957 ts_tablet_manager.cc:1905] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:17.489423 4961 ts_tablet_manager.cc:1428] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 01:55:17.494378 4959 consensus_queue.cc:260] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.495163 4957 ts_tablet_manager.cc:1918] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:17.495275 4957 log.cc:1199] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/wals/65637e7e27e44ed7ace78227be5e4a50
I20250904 01:55:17.495714 4957 ts_tablet_manager.cc:1939] T 65637e7e27e44ed7ace78227be5e4a50 P 5b23e87ab60248f391fb17326f5e3934: Deleting consensus metadata
I20250904 01:55:17.496568 4959 ts_tablet_manager.cc:1428] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250904 01:55:17.497094 4956 ts_tablet_manager.cc:1918] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:17.497218 4956 log.cc:1199] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/wals/4f9d1156f12b43839c00ac99ffdb118e
I20250904 01:55:17.497574 4959 tablet_bootstrap.cc:492] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:17.497612 4956 ts_tablet_manager.cc:1939] T 4f9d1156f12b43839c00ac99ffdb118e P 7d96d300478849e3b11402fbd34fef15: Deleting consensus metadata
I20250904 01:55:17.499102 4959 tablet_bootstrap.cc:654] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:17.500612 4957 tablet_replica.cc:331] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: stopping tablet replica
I20250904 01:55:17.500788 4957 raft_consensus.cc:2241] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:17.500946 4957 raft_consensus.cc:2270] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:17.501400 4957 ts_tablet_manager.cc:1905] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:17.506843 4959 tablet_bootstrap.cc:492] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:17.508325 4959 ts_tablet_manager.cc:1397] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250904 01:55:17.508998 4959 raft_consensus.cc:357] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.509171 4959 raft_consensus.cc:383] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:17.509248 4959 raft_consensus.cc:738] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:17.509508 4959 consensus_queue.cc:260] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.507119 4957 ts_tablet_manager.cc:1918] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:17.511857 4957 log.cc:1199] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/wals/4f9d1156f12b43839c00ac99ffdb118e
I20250904 01:55:17.512122 4959 ts_tablet_manager.cc:1428] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 01:55:17.512217 4957 ts_tablet_manager.cc:1939] T 4f9d1156f12b43839c00ac99ffdb118e P 5b23e87ab60248f391fb17326f5e3934: Deleting consensus metadata
I20250904 01:55:17.654767 4835 raft_consensus.cc:491] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:17.654965 4835 raft_consensus.cc:513] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.655808 4835 leader_election.cc:290] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:17.656545 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b299dd322a4c35bbed9ec90e9fa3b6" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:17.656564 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b299dd322a4c35bbed9ec90e9fa3b6" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:17.656764 4256 raft_consensus.cc:2466] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:17.656798 4120 raft_consensus.cc:2466] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:17.657253 3917 leader_election.cc:304] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:17.657554 4835 raft_consensus.cc:2802] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:17.657663 4835 raft_consensus.cc:491] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:17.657749 4835 raft_consensus.cc:3058] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:17.660621 4835 raft_consensus.cc:513] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.661558 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b299dd322a4c35bbed9ec90e9fa3b6" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:17.661660 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b299dd322a4c35bbed9ec90e9fa3b6" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:17.661757 4120 raft_consensus.cc:3058] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:17.661808 4256 raft_consensus.cc:3058] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:17.664247 4120 raft_consensus.cc:2466] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:17.664307 4256 raft_consensus.cc:2466] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
I20250904 01:55:17.664693 3916 leader_election.cc:304] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:17.665481 4835 leader_election.cc:290] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:17.665643 4974 raft_consensus.cc:2802] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:17.665814 4974 raft_consensus.cc:695] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:17.666164 4974 consensus_queue.cc:237] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:17.685993 4974 raft_consensus.cc:491] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:17.686187 4974 raft_consensus.cc:513] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.686877 4974 leader_election.cc:290] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:17.687309 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac85bb6dc61f4d0cbe26f94b4e5c0f96" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:17.687409 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac85bb6dc61f4d0cbe26f94b4e5c0f96" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15" is_pre_election: true
I20250904 01:55:17.687511 4120 raft_consensus.cc:2466] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:17.687594 4256 raft_consensus.cc:2466] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 0.
I20250904 01:55:17.687970 3916 leader_election.cc:304] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
W20250904 01:55:17.691435 4056 connection.cc:537] server connection from 127.0.0.1:52476 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:17.691720 4190 connection.cc:537] server connection from 127.0.0.1:59176 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:17.691953 4189 connection.cc:537] client connection to 127.3.77.60:38761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:17.693275 3915 connection.cc:537] server connection from 127.0.0.1:55230 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:17.693550 3915 connection.cc:537] client connection to 127.3.77.60:38761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250904 01:55:17.695652 4974 raft_consensus.cc:2802] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:17.695789 4974 raft_consensus.cc:491] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:17.695966 4974 raft_consensus.cc:3058] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:17.698868 4974 raft_consensus.cc:513] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
W20250904 01:55:17.699854 4053 connection.cc:537] client connection to 127.3.77.60:38761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250904 01:55:17.701870 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac85bb6dc61f4d0cbe26f94b4e5c0f96" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:17.702044 4120 raft_consensus.cc:3058] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
W20250904 01:55:17.702625 4308 connection.cc:537] client connection to 127.3.77.60:38761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:17.703140 4308 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.60:38761)
W20250904 01:55:17.703599 4308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.60:38761)
W20250904 01:55:17.704166 4308 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.60:38761)
I20250904 01:55:17.704495 4120 raft_consensus.cc:2466] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
W20250904 01:55:17.704558 4308 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.60:38761)
I20250904 01:55:17.705163 3917 leader_election.cc:304] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934; no voters:
I20250904 01:55:17.705440 4835 raft_consensus.cc:2802] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:17.709537 4974 leader_election.cc:290] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
W20250904 01:55:17.709851 4932 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:17.692643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.60:38761 (local address 127.3.77.3:59029)
0904 01:55:17.692880 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:17.692898 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:17.693046 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:17.709558 (+ 16512us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125}
I20250904 01:55:17.710036 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac85bb6dc61f4d0cbe26f94b4e5c0f96" candidate_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96d300478849e3b11402fbd34fef15"
I20250904 01:55:17.710247 4256 raft_consensus.cc:3058] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
W20250904 01:55:17.711738 4827 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:17.698465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.60:38761 (local address 127.3.77.1:36083)
0904 01:55:17.698661 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:17.698680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:17.698779 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:17.710693 (+ 11914us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":98}
W20250904 01:55:17.712303 4828 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:17.702049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.60:38761 (local address 127.3.77.2:60907)
0904 01:55:17.702275 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:17.702291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:17.702386 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:17.712120 (+ 9734us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":108}
I20250904 01:55:17.712721 4256 raft_consensus.cc:2466] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca66c5fca0c4305a2857a0074a8f8cf in term 1.
W20250904 01:55:17.713089 4450 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:17.706283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.60:38761 (local address 127.0.0.1:54550)
0904 01:55:17.706843 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:17.706859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:17.707010 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:17.712751 (+ 5741us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":3876,"threads_started":1}
I20250904 01:55:17.713075 4835 raft_consensus.cc:695] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Running, Role: LEADER
I20250904 01:55:17.713421 4835 consensus_queue.cc:237] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } }
I20250904 01:55:17.919566 4979 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7f298ba82a5e49028407388810de4b71)
I20250904 01:55:17.919731 4979 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:17.921159 4979 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:17.921887 4851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:17.923003 4850 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:17.923168 4851 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:17.923323 4851 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:17.923573 4979 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 01:55:18.147819 3982 raft_consensus.cc:1215] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250904 01:55:18.592097 4835 consensus_queue.cc:1035] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:18.597028 4835 consensus_queue.cc:1035] T 04b299dd322a4c35bbed9ec90e9fa3b6 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:18.622478 4994 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7f298ba82a5e49028407388810de4b71)
I20250904 01:55:18.622609 4994 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:18.623154 4994 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:18.624701 4851 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:18.624891 4850 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:18.625028 4850 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:18.625226 4994 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 01:55:18.650238 4835 consensus_queue.cc:1035] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:18.653627 4988 consensus_queue.cc:1035] T ac85bb6dc61f4d0cbe26f94b4e5c0f96 P 2ca66c5fca0c4305a2857a0074a8f8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:19.322729 5006 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7f298ba82a5e49028407388810de4b71)
I20250904 01:55:19.322887 5006 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:19.323349 5006 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:19.325038 4851 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:19.325206 4850 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:19.325347 4850 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:19.325565 5006 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 01:55:20.355836 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--webserver_interface=127.3.77.61
--webserver_port=46249
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:20.644218 5024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:20.644495 5024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:20.644605 5024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:20.652179 5024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:20.652258 5024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:20.652321 5024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:20.652356 5024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:20.660785 5024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.77.61
--webserver_port=46249
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:20.662544 5024 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:20.665057 5024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:20.672606 5036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:20.672575 5037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:20.673221 5039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:20.673831 5024 server_base.cc:1047] running on GCE node
I20250904 01:55:20.674607 5024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:20.676090 5024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:20.677309 5024 hybrid_clock.cc:648] HybridClock initialized: now 1756950920677262 us; error 37 us; skew 500 ppm
I20250904 01:55:20.677654 5024 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:20.679903 5024 webserver.cc:480] Webserver started at http://127.3.77.61:46249/ using document root <none> and password file <none>
I20250904 01:55:20.680631 5024 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:20.680742 5024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:20.686170 5024 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20250904 01:55:20.689098 5045 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:20.690147 5024 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250904 01:55:20.690301 5024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
uuid: "6ef804bcecd943d89c7af93e9375b1f3"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:20.690869 5024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:20.705735 5024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:20.706328 5024 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:20.706525 5024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:20.721974 5024 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.61:36307
I20250904 01:55:20.722004 5096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.61:36307 every 8 connection(s)
I20250904 01:55:20.723237 5024 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
I20250904 01:55:20.728560 5097 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:20.729947 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 5024
I20250904 01:55:20.733939 5097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap starting.
I20250904 01:55:20.749938 5097 log.cc:826] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:20.772756 5102 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7f298ba82a5e49028407388810de4b71)
I20250904 01:55:20.772922 5102 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:20.773468 5102 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:20.777000 4850 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:20.777340 5097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=32 ignored=0} inserts{seen=23 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250904 01:55:20.777807 5097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap complete.
I20250904 01:55:20.782507 5097 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 3 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:20.785617 5097 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef804bcecd943d89c7af93e9375b1f3, State: Initialized, Role: FOLLOWER
I20250904 01:55:20.786188 5097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:20.786923 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:20.787192 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:20.787770 5097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:20.787824 5072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:20.788178 5072 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 3.
I20250904 01:55:20.788625 4851 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:20.788887 5102 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250904 01:55:20.789028 5102 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7f298ba82a5e49028407388810de4b71)
I20250904 01:55:20.789116 5102 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 5 FOLLOWER]: Advancing to term 6
I20250904 01:55:20.792033 5102 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:20.792544 5102 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 election: Requested vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:20.793165 5072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3"
I20250904 01:55:20.793354 5072 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 3 FOLLOWER]: Advancing to term 6
W20250904 01:55:20.795238 4850 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:20.807224 5072 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 6.
I20250904 01:55:20.807817 4851 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:20.808218 5102 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Leader election won for term 6
I20250904 01:55:20.808737 5102 raft_consensus.cc:695] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 LEADER]: Becoming Leader. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Running, Role: LEADER
I20250904 01:55:20.809180 5102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 5.43, Last appended by leader: 43, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:20.821609 5121 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:20.821712 5121 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: loading cluster ID for follower catalog manager: success
I20250904 01:55:20.826568 5121 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: acquiring CA information for follower catalog manager: success
I20250904 01:55:20.829706 5121 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:20.832544 5125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 6 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:20.832823 5125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:20.833577 5128 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:20.835896 5128 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:20.836339 5128 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:20.836655 5128 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:20.837018 5128 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:20.837419 5128 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:20.837898 5128 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:20.838311 5128 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:20.838709 5128 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:20.839119 5128 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:20.840610 5128 tablet_loader.cc:96] loaded metadata for tablet 04b299dd322a4c35bbed9ec90e9fa3b6 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:20.840916 5128 tablet_loader.cc:96] loaded metadata for tablet 138000c0bebd4169a7d2db4a7f660845 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:20.841357 5128 tablet_loader.cc:96] loaded metadata for tablet 30378e1466de4f559ac840b676310486 (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:20.841625 5128 tablet_loader.cc:96] loaded metadata for tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:20.842006 5128 tablet_loader.cc:96] loaded metadata for tablet 5742132feece4a58a803d0769ee51d05 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:20.842557 5128 tablet_loader.cc:96] loaded metadata for tablet 9f05d628c97e4f57ad4cf4337fe6810f (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:20.842811 5128 tablet_loader.cc:96] loaded metadata for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:20.843434 5128 tablet_loader.cc:96] loaded metadata for tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:20.843710 5128 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:20.843930 5128 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:20.845279 5128 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:20.845391 5128 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:20.846915 5128 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:20.848086 5128 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Loaded TSK: 0
I20250904 01:55:20.848868 5128 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:20.849658 5128 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:20.850657 5128 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:20.852880 5024 master_runner.cc:186] Error getting master registration for 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111),
I20250904 01:55:20.856051 5024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:20.916769 4865 master_service.cc:948] ReplaceTablet: received request to replace tablet 30378e1466de4f559ac840b676310486 from {username='slave'} at 127.0.0.1:46654
I20250904 01:55:20.918686 4866 master_service.cc:948] ReplaceTablet: received request to replace tablet 5742132feece4a58a803d0769ee51d05 from {username='slave'} at 127.0.0.1:46654
I20250904 01:55:20.918781 4861 master_service.cc:948] ReplaceTablet: received request to replace tablet ba52aec4f85c4723b600ab9271beb725 from {username='slave'} at 127.0.0.1:46654
I20250904 01:55:20.917933 4860 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46654:
name: "default.Table_338af7b0a5ed48918a49777a432224f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:20.919023 4859 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46654:
name: "default.Table_d723b3eb51ff4772a7331c9338cb1918"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:20.921684 4863 master_service.cc:948] ReplaceTablet: received request to replace tablet 04b299dd322a4c35bbed9ec90e9fa3b6 from {username='slave'} at 127.0.0.1:46654
I20250904 01:55:20.921995 4857 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46654:
name: "default.Table_968a17a9054a48dda6494188adb73474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:20.922557 4862 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46654:
name: "default.Table_d03c97312adf4523be9539809e8fdb5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:20.923867 4861 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46654:
name: "default.Table_c245a71ea8e74145959886fcfac66f07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:20.924870 4858 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46654:
name: "default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:20.925411 4864 master_service.cc:948] ReplaceTablet: received request to replace tablet 9f05d628c97e4f57ad4cf4337fe6810f from {username='slave'} at 127.0.0.1:46654
F20250904 01:55:20.929733 4918 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250904 01:55:20.931023 5072 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Refusing update from remote peer 29b8a62cc8d04df781f991da2bd83ccc: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 6 index: 45. (index mismatch)
I20250904 01:55:20.931828 5125 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 32, Time since last communication: 0.000s
@ 0x7fac4d0c7dcd google::LogMessage::Fail() at ??:0
@ 0x7fac4d0cbb93 google::LogMessage::SendToLog() at ??:0
W20250904 01:55:20.933724 4850 consensus_peers.cc:489] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc -> Peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Couldn't send request to peer 7f298ba82a5e49028407388810de4b71. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fac4d0c77cc google::LogMessage::Flush() at ??:0
@ 0x7fac4d0c7a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fac6350a7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fac63509061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fac63510cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fac63510b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fac4fe302b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fac532546db start_thread at ??:0
@ 0x7fac4bf7871f clone at ??:0
I20250904 01:55:20.982924 5106 mvcc.cc:204] Tried to move back new op lower bound from 7196470957690916864 to 7196470944366325760. Current Snapshot: MvccSnapshot[applied={T|T < 7196470957690916864 or (T in {7196470957690916864})}]
I20250904 01:55:20.984412 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 6 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:20.984651 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:20.984652 5133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:20.984947 5133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:20.984973 5144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:20.985270 5144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:20.985103 5141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:20.985383 5141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:21.095894 4030 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:21.100498 5062 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:47417; Asking this server to re-register.
I20250904 01:55:21.101469 4030 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:21.103241 5062 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
W20250904 01:55:21.154096 5047 connection.cc:537] server connection from 127.0.0.1:37830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:21.154608 3915 connection.cc:537] client connection to 127.3.77.62:34273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:21.155683 4308 connection.cc:537] client connection to 127.3.77.62:34273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:21.156306 4308 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.62:34273)
W20250904 01:55:21.156520 4308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:21.156697 4308 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:21.157078 4308 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.62:34273)
I20250904 01:55:21.176445 4168 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:21.179577 5061 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:49217; Asking this server to re-register.
I20250904 01:55:21.180655 4168 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:21.183473 5061 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:21.184634 4303 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:21.186556 5062 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:52481; Asking this server to re-register.
I20250904 01:55:21.187187 4303 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:21.188536 5062 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:21.435637 5141 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:21.435783 5141 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:21.437353 5141 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:21.438341 5046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:21.439677 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:21.439880 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:21.439963 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:21.440215 5141 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250904 01:55:22.168556 5159 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:22.168715 5159 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:22.169324 5159 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:22.171321 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:22.171518 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:22.171617 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:22.171887 5159 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250904 01:55:22.884959 5169 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:22.885133 5169 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:22.885730 5169 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:22.887951 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:22.888113 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:22.888203 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:22.888399 5169 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250904 01:55:23.877866 5181 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:23.878047 5181 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:23.878624 5181 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:23.880641 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:23.880807 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:23.880910 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:23.881165 5181 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250904 01:55:24.000437 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--webserver_interface=127.3.77.62
--webserver_port=46147
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:24.309095 5185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:24.309358 5185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:24.309465 5185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:24.317836 5185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:24.317934 5185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:24.318022 5185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:24.318086 5185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:24.328338 5185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.77.62
--webserver_port=46147
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:24.329705 5185 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:24.331379 5185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:24.338474 5196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:24.339408 5198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:24.339461 5195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:24.339838 5185 server_base.cc:1047] running on GCE node
I20250904 01:55:24.340495 5185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:24.341655 5185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:24.342833 5185 hybrid_clock.cc:648] HybridClock initialized: now 1756950924342797 us; error 27 us; skew 500 ppm
I20250904 01:55:24.343119 5185 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:24.345044 5185 webserver.cc:480] Webserver started at http://127.3.77.62:46147/ using document root <none> and password file <none>
I20250904 01:55:24.345686 5185 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:24.345782 5185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:24.350270 5185 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:55:24.352775 5204 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:24.353821 5185 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250904 01:55:24.353933 5185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:24.354602 5185 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:24.366707 5185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:24.367326 5185 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:24.367540 5185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:24.385843 5185 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.62:34273
I20250904 01:55:24.385864 5255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.62:34273 every 8 connection(s)
I20250904 01:55:24.387310 5185 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
I20250904 01:55:24.393231 5256 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:24.396160 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 5185
I20250904 01:55:24.396462 5256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap starting.
I20250904 01:55:24.408418 5256 log.cc:826] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:24.439823 5256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=43 ignored=0} inserts{seen=28 ignored=0} mutations{seen=30 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20250904 01:55:24.440369 5256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap complete.
I20250904 01:55:24.445375 5256 raft_consensus.cc:357] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:24.447908 5256 raft_consensus.cc:738] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Initialized, Role: FOLLOWER
I20250904 01:55:24.448514 5256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 6.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:24.449237 5261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:24.449460 5261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:24.450085 5256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:24.460913 5275 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:24.461050 5275 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: loading cluster ID for follower catalog manager: success
I20250904 01:55:24.466650 5275 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: acquiring CA information for follower catalog manager: success
I20250904 01:55:24.469476 5275 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:24.480473 5185 master_runner.cc:186] Error getting master registration for 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111),
I20250904 01:55:24.483562 5185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:24.669922 4029 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:24.672932 5221 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:43719; Asking this server to re-register.
I20250904 01:55:24.673673 4029 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:24.675153 5221 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:24.708392 4302 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:24.709514 5221 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:52323; Asking this server to re-register.
I20250904 01:55:24.710074 4302 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:24.711663 5221 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:24.713871 4167 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:24.715067 5221 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:36503; Asking this server to re-register.
I20250904 01:55:24.715615 4167 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:24.716857 5221 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:25.005136 5285 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:25.005391 5285 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:25.007073 5285 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:25.007532 5072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 7 candidate_status { last_received { term: 6 index: 48 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:25.007711 5072 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 6.
I20250904 01:55:25.008198 5206 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters:
I20250904 01:55:25.008502 5285 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250904 01:55:25.008613 5285 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:25.008703 5285 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 6 FOLLOWER]: Advancing to term 7
I20250904 01:55:25.011708 5285 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:25.012245 5285 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 7 election: Requested vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:25.012589 5072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 7 candidate_status { last_received { term: 6 index: 48 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3"
I20250904 01:55:25.012749 5072 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 6 FOLLOWER]: Advancing to term 7
I20250904 01:55:25.015292 5072 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 7.
I20250904 01:55:25.016242 5206 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters:
I20250904 01:55:25.016505 5285 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 7 FOLLOWER]: Leader election won for term 7
I20250904 01:55:25.016871 5285 raft_consensus.cc:695] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 7 LEADER]: Becoming Leader. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Running, Role: LEADER
I20250904 01:55:25.017237 5285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 6.48, Last appended by leader: 48, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:25.019814 5287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 7 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:25.020098 5287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
W20250904 01:55:25.022792 5205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:25.023975 5205 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:25.024359 5205 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:25.492157 5205 consensus_peers.cc:489] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc -> Peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Couldn't send request to peer 7f298ba82a5e49028407388810de4b71. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:55:25.638623 5072 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Refusing update from remote peer 29b8a62cc8d04df781f991da2bd83ccc: Log matching property violated. Preceding OpId in replica: term: 6 index: 48. Preceding OpId from leader: term: 7 index: 49. (index mismatch)
I20250904 01:55:25.639299 5287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 43, Time since last communication: 0.000s
I20250904 01:55:25.641191 5292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 7 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:25.641373 5292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:25.647608 5302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:25.647818 5292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:25.647964 5292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:25.647946 5302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:25.648502 5287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:25.648722 5287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:25.648766 5294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:25.649048 5294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:25.665246 5289 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:25.667769 5289 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:25.668188 5289 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:25.668533 5289 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:25.668924 5289 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:25.669299 5289 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:25.669678 5289 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:25.669982 5289 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:25.670373 5289 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:25.670701 5289 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:25.672412 5289 tablet_loader.cc:96] loaded metadata for tablet 138000c0bebd4169a7d2db4a7f660845 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:25.673020 5289 tablet_loader.cc:96] loaded metadata for tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:25.673609 5289 tablet_loader.cc:96] loaded metadata for tablet 5b42841c537742dab8de6bec2dcbcf29 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:25.674140 5289 tablet_loader.cc:96] loaded metadata for tablet 99c7b2028e1f42e99074eee61163dbd9 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:25.674783 5289 tablet_loader.cc:96] loaded metadata for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:25.675622 5289 tablet_loader.cc:96] loaded metadata for tablet c0d920867c2c426586568745632c810c (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:25.676079 5289 tablet_loader.cc:96] loaded metadata for tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:25.676314 5289 tablet_loader.cc:96] loaded metadata for tablet eea8675339d24616b52d0c53c8f48a6f (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:25.676625 5289 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:25.676827 5289 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:25.677719 5289 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:25.677795 5289 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:25.679442 5289 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:25.680326 5289 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Loaded TSK: 0
I20250904 01:55:25.681092 5289 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:25.681962 5289 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:25.682840 5289 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:25.741442 5221 master_service.cc:948] ReplaceTablet: received request to replace tablet 30378e1466de4f559ac840b676310486 from {username='slave'} at 127.0.0.1:55996
I20250904 01:55:25.741922 5220 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55996:
name: "default.Table_39d410e70d9349abad85c4dccbad54bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:25.742655 5219 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b" } modify_external_catalogs: true
I20250904 01:55:25.742810 5218 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:25.742877 5215 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_5a0ac2833a3d4a2b98f538f7416ca2d1" } new_table_name: "default.Table_ae25e811a20d449d8d8c57cca63f758b" modify_external_catalogs: true
I20250904 01:55:25.742962 5219 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b" } modify_external_catalogs: true
I20250904 01:55:25.743054 5218 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:25.742687 5214 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_18be6a726bc64f8c8efad17c700b66aa" } new_table_name: "default.Table_db0617ec7a994e45b7e7ed190308ddbe" modify_external_catalogs: true
I20250904 01:55:25.743465 5217 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:25.743378 5216 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55996:
name: "default.Table_bce07055a7064408b762f1534d59a90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:25.743698 5217 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:25.744117 5221 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55996:
name: "default.Table_bc5b38d3266841d6b4a43d4983cff6b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:25.744335 5213 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55996:
name: "default.Table_d03c97312adf4523be9539809e8fdb5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:25.744560 5212 master_service.cc:948] ReplaceTablet: received request to replace tablet 04b299dd322a4c35bbed9ec90e9fa3b6 from {username='slave'} at 127.0.0.1:55996
I20250904 01:55:25.745143 5212 master_service.cc:948] ReplaceTablet: received request to replace tablet 9f05d628c97e4f57ad4cf4337fe6810f from {username='slave'} at 127.0.0.1:55996
I20250904 01:55:25.745518 5212 master_service.cc:948] ReplaceTablet: received request to replace tablet 30378e1466de4f559ac840b676310486 from {username='slave'} at 127.0.0.1:55996
I20250904 01:55:25.745889 5212 master_service.cc:948] ReplaceTablet: received request to replace tablet 5742132feece4a58a803d0769ee51d05 from {username='slave'} at 127.0.0.1:55996
I20250904 01:55:25.746402 5212 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55996:
name: "default.Table_338af7b0a5ed48918a49777a432224f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250904 01:55:25.748983 5274 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fab47440dcd google::LogMessage::Fail() at ??:0
@ 0x7fab47444b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fab474407cc google::LogMessage::Flush() at ??:0
@ 0x7fab47440a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fab5d8837b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fab5d882061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fab5d889cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fab5d889b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fab4a1a92b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fab4d5cd6db start_thread at ??:0
@ 0x7fab462f171f clone at ??:0
W20250904 01:55:25.918915 4308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:25.919147 4308 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:25.919309 4308 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:25.919579 5319 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:25.856769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.77.62:34273 (local address 127.0.0.1:56016)
0904 01:55:25.857285 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0904 01:55:25.857300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0904 01:55:25.857407 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 01:55:25.918465 (+ 61058us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":294,"threads_started":1}
I20250904 01:55:25.919852 5024 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250904 01:55:25.920027 5024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:55:25.950639 4308 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.62:34273)
I20250904 01:55:26.143831 5302 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:26.144021 5302 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:26.144731 5302 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:26.146602 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:26.146782 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:26.146865 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:26.147049 5302 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250904 01:55:26.216403 4302 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
I20250904 01:55:26.220855 4167 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
I20250904 01:55:26.797397 5327 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:26.797529 5327 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:26.798084 5327 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:26.798784 5046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250904 01:55:26.800042 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:26.800248 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:26.800331 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:26.800542 5327 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250904 01:55:26.982870 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--webserver_interface=127.3.77.62
--webserver_port=46147
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:27.280102 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:27.280357 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:27.280463 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:27.288239 5331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:27.288313 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:27.288386 5331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:27.288437 5331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:27.296758 5331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.77.62
--webserver_port=46147
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:27.298070 5331 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:27.299535 5331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:27.306177 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:27.306433 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:27.307339 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:27.307781 5331 server_base.cc:1047] running on GCE node
I20250904 01:55:27.308360 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:27.309401 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:27.310609 5331 hybrid_clock.cc:648] HybridClock initialized: now 1756950927310567 us; error 32 us; skew 500 ppm
I20250904 01:55:27.310894 5331 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:27.312683 5331 webserver.cc:480] Webserver started at http://127.3.77.62:46147/ using document root <none> and password file <none>
I20250904 01:55:27.313347 5331 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:27.313439 5331 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:27.318571 5331 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:55:27.320967 5347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:27.322018 5331 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250904 01:55:27.322155 5331 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:27.322754 5331 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:27.348608 5331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:27.349188 5331 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:27.349422 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:27.366888 5331 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.62:34273
I20250904 01:55:27.366918 5398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.62:34273 every 8 connection(s)
I20250904 01:55:27.368180 5331 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
I20250904 01:55:27.368741 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 5331
I20250904 01:55:27.375582 5399 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:27.379807 5399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap starting.
I20250904 01:55:27.390529 5399 log.cc:826] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:27.421788 5399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=32 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:55:27.422298 5399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap complete.
I20250904 01:55:27.427920 5399 raft_consensus.cc:357] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:27.428452 5399 raft_consensus.cc:738] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Initialized, Role: FOLLOWER
I20250904 01:55:27.428946 5399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 7.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:27.429575 5404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:27.429788 5404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:27.430339 5399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:27.441327 5418 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:27.446290 5418 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: loading cluster ID for follower catalog manager: success
I20250904 01:55:27.450289 5418 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: acquiring CA information for follower catalog manager: success
I20250904 01:55:27.452669 5418 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:27.465829 5331 master_runner.cc:186] Error getting master registration for 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111),
I20250904 01:55:27.469107 5331 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:27.687762 4029 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:27.692297 5363 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:59815; Asking this server to re-register.
I20250904 01:55:27.693109 4029 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:27.694921 5363 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:27.727869 4302 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:27.728076 4302 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
I20250904 01:55:27.730594 5364 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:34533; Asking this server to re-register.
I20250904 01:55:27.731350 4302 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:27.732753 5364 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:27.735834 4167 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:27.736008 4167 heartbeater.cc:499] Master 127.3.77.62:34273 was elected leader, sending a full tablet report...
I20250904 01:55:27.737747 5363 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:47309; Asking this server to re-register.
I20250904 01:55:27.738384 4167 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:27.739614 5363 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:27.827913 5423 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:27.828120 5423 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:27.828784 5423 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:27.831028 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:27.834502 5374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 8 candidate_status { last_received { term: 7 index: 49 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
I20250904 01:55:27.834956 5374 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef804bcecd943d89c7af93e9375b1f3 in term 7.
I20250904 01:55:27.835409 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:27.835657 5423 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250904 01:55:27.835780 5423 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:27.835884 5423 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 7 FOLLOWER]: Advancing to term 8
I20250904 01:55:27.837700 5423 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:27.838093 5423 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 election: Requested vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:27.838446 5374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 8 candidate_status { last_received { term: 7 index: 49 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc"
I20250904 01:55:27.838604 5374 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 7 FOLLOWER]: Advancing to term 8
W20250904 01:55:27.839249 5046 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:27.842168 5374 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef804bcecd943d89c7af93e9375b1f3 in term 8.
I20250904 01:55:27.842527 5046 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:27.842799 5423 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Leader election won for term 8
I20250904 01:55:27.843169 5423 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 LEADER]: Becoming Leader. State: Replica: 6ef804bcecd943d89c7af93e9375b1f3, State: Running, Role: LEADER
I20250904 01:55:27.843540 5423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 7.49, Last appended by leader: 49, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:27.844882 5427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef804bcecd943d89c7af93e9375b1f3. Latest consensus state: current_term: 8 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:27.845211 5427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:27.845794 5429 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:27.848150 5429 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:27.848562 5429 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:27.848886 5429 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:27.849188 5429 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:27.849493 5429 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:27.849790 5429 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:27.850066 5429 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:27.850401 5429 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:27.850694 5429 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:27.852272 5429 tablet_loader.cc:96] loaded metadata for tablet 138000c0bebd4169a7d2db4a7f660845 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:27.852834 5429 tablet_loader.cc:96] loaded metadata for tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:27.853638 5429 tablet_loader.cc:96] loaded metadata for tablet 5b42841c537742dab8de6bec2dcbcf29 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:27.854380 5429 tablet_loader.cc:96] loaded metadata for tablet 99c7b2028e1f42e99074eee61163dbd9 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:27.854811 5429 tablet_loader.cc:96] loaded metadata for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:27.855360 5429 tablet_loader.cc:96] loaded metadata for tablet c0d920867c2c426586568745632c810c (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:27.855742 5429 tablet_loader.cc:96] loaded metadata for tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:27.855960 5429 tablet_loader.cc:96] loaded metadata for tablet eea8675339d24616b52d0c53c8f48a6f (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:27.856233 5429 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:27.856426 5429 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:27.857339 5429 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:27.857411 5429 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:27.858569 5429 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:27.859375 5429 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Loaded TSK: 0
I20250904 01:55:27.860069 5429 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:27.860852 5429 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:27.861608 5429 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:27.920493 5062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37810:
table { table_name: "default.Table_7d5858a140ad4b7eb83927265abad9fe" } modify_external_catalogs: true
I20250904 01:55:27.920419 5061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37810:
name: "default.Table_782e3736e3434cafa6e2825f14c5900b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:27.920756 5062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37810:
table { table_name: "default.Table_7d5858a140ad4b7eb83927265abad9fe" } modify_external_catalogs: true
I20250904 01:55:27.921240 5054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37810:
name: "default.Table_bce07055a7064408b762f1534d59a90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:27.921463 5060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37810:
table { table_name: "default.Table_e916036e5adf47c886c74e8088132031" } new_table_name: "default.Table_386e7bb279394e4ba14a48582a9ba8f8" modify_external_catalogs: true
I20250904 01:55:27.922088 5059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37810:
table { table_name: "default.Table_9a196bab16bb4604ab9b00d45335f022" } new_table_name: "default.Table_09793f14275347b0aaa65279c0b3bc41" modify_external_catalogs: true
I20250904 01:55:27.922151 5055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37810:
name: "default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:27.922646 5053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37810:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:27.922896 5056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37810:
name: "default.Table_968a17a9054a48dda6494188adb73474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:27.922907 5053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37810:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:27.923280 5058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37810:
table { table_name: "default.Table_a524c1ec64554d8eb7687bfa84e4579c" } new_table_name: "default.Table_3a626a942a5e41e9b5d523f0be9b7387" modify_external_catalogs: true
I20250904 01:55:27.923550 5057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37810:
name: "default.Table_c245a71ea8e74145959886fcfac66f07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250904 01:55:27.927006 5120 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7ff1cc55edcd google::LogMessage::Fail() at ??:0
@ 0x7ff1cc562b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff1cc55e7cc google::LogMessage::Flush() at ??:0
@ 0x7ff1cc55ea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ff1e29a17b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7ff1e29a0061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ff1e29a7cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7ff1e29a7b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7ff1cf2c72b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff1d26eb6db start_thread at ??:0
@ 0x7ff1cb40f71f clone at ??:0
W20250904 01:55:28.108197 4309 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.61:36307)
W20250904 01:55:28.108729 4309 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.61:36307)
W20250904 01:55:28.109174 4309 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.61:36307)
W20250904 01:55:28.109475 4309 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.61:36307)
I20250904 01:55:28.591800 5443 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:28.591969 5443 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:28.593536 5443 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:28.594362 5349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:28.595588 5349 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:28.595808 5348 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:28.595919 5348 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:28.596259 5443 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250904 01:55:29.182468 5447 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:29.182603 5447 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:29.183112 5447 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:29.184769 5348 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:29.184927 5349 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:29.185098 5349 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:29.185333 5447 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250904 01:55:29.726625 3380 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu pid 5331
I20250904 01:55:29.736433 5394 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 01:55:29.831717 3380 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu with pid 5331
I20250904 01:55:30.784668 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--webserver_interface=127.3.77.62
--webserver_port=46147
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:31.080178 5457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:31.080425 5457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:31.080492 5457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:31.088115 5457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:31.088183 5457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:31.088239 5457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:31.088284 5457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:31.096794 5457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.62:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.77.62
--webserver_port=46147
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:31.098145 5457 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:31.099709 5457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:31.106748 5463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:31.106706 5462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:31.106773 5465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:31.107424 5457 server_base.cc:1047] running on GCE node
I20250904 01:55:31.108001 5457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:31.109061 5457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:31.110260 5457 hybrid_clock.cc:648] HybridClock initialized: now 1756950931110190 us; error 41 us; skew 500 ppm
I20250904 01:55:31.110551 5457 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:31.112321 5457 webserver.cc:480] Webserver started at http://127.3.77.62:46147/ using document root <none> and password file <none>
I20250904 01:55:31.113108 5457 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:31.113212 5457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:31.118764 5457 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250904 01:55:31.121588 5471 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:31.122923 5457 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:55:31.123046 5457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
uuid: "29b8a62cc8d04df781f991da2bd83ccc"
format_stamp: "Formatted at 2025-09-04 01:55:05 on dist-test-slave-ch6q"
I20250904 01:55:31.123589 5457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:31.147850 5457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:31.148423 5457 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:31.148628 5457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:31.165123 5457 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.62:34273
I20250904 01:55:31.165179 5522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.62:34273 every 8 connection(s)
I20250904 01:55:31.166638 5457 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-0/data/info.pb
I20250904 01:55:31.169955 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 5457
I20250904 01:55:31.173096 5523 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:31.176237 5523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap starting.
I20250904 01:55:31.187218 5523 log.cc:826] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:31.209360 4029 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:31.221215 5523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=32 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:55:31.221735 5523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Bootstrap complete.
I20250904 01:55:31.226202 5523 raft_consensus.cc:357] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:31.226783 5523 raft_consensus.cc:738] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Initialized, Role: FOLLOWER
I20250904 01:55:31.227226 5523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 7.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:31.227828 5527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:31.228047 5527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:31.228654 5523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:31.240849 5541 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:31.241001 5541 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: loading cluster ID for follower catalog manager: success
I20250904 01:55:31.247308 5541 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: acquiring CA information for follower catalog manager: success
I20250904 01:55:31.250931 5541 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:31.248813 5457 master_runner.cc:186] Error getting master registration for 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111),
I20250904 01:55:31.253970 5457 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:31.360220 4167 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:31.360711 4302 heartbeater.cc:344] Connected to a master server at 127.3.77.62:34273
I20250904 01:55:31.362699 5488 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:37067; Asking this server to re-register.
I20250904 01:55:31.362677 5487 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:51687; Asking this server to re-register.
I20250904 01:55:31.363288 4167 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:31.363312 4302 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:31.365164 5487 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:31.365267 5488 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:31.712266 5488 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:43113; Asking this server to re-register.
I20250904 01:55:31.712795 4029 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:31.713893 5488 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:31.827390 5546 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:31.827646 5546 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:31.829250 5546 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:31.830377 5473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:31.831386 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:55:31.831555 5473 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
I20250904 01:55:31.831696 5473 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:31.831960 5546 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250904 01:55:32.348690 5551 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:32.348835 5551 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:32.349421 5551 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:32.351147 5473 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:32.351293 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:32.351423 5472 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:32.351655 5551 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250904 01:55:33.183029 5556 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:33.183167 5556 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:33.183625 5556 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:33.185345 5473 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:33.185477 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:33.185617 5472 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:33.185833 5556 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250904 01:55:33.992673 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--webserver_interface=127.3.77.61
--webserver_port=46249
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:34.285358 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:34.285595 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:34.285725 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:34.292585 5560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:34.292650 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:34.292697 5560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:34.292732 5560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:34.301085 5560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.77.61
--webserver_port=46249
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:34.302362 5560 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:34.303828 5560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:34.311028 5566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:34.311069 5569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:34.311089 5567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:34.311996 5560 server_base.cc:1047] running on GCE node
I20250904 01:55:34.312609 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:34.313752 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:34.314930 5560 hybrid_clock.cc:648] HybridClock initialized: now 1756950934314886 us; error 36 us; skew 500 ppm
I20250904 01:55:34.315219 5560 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:34.317205 5560 webserver.cc:480] Webserver started at http://127.3.77.61:46249/ using document root <none> and password file <none>
I20250904 01:55:34.317842 5560 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:34.317934 5560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:34.322139 5560 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:55:34.324549 5575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:34.325600 5560 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250904 01:55:34.325747 5560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
uuid: "6ef804bcecd943d89c7af93e9375b1f3"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:34.326359 5560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:34.353186 5560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:34.353819 5560 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:34.354076 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:34.370838 5560 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.61:36307
I20250904 01:55:34.370869 5626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.61:36307 every 8 connection(s)
I20250904 01:55:34.372133 5560 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
I20250904 01:55:34.377931 5627 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:34.378914 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 5560
I20250904 01:55:34.381377 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap starting.
I20250904 01:55:34.393440 5627 log.cc:826] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:34.426626 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=49 ignored=0} inserts{seen=32 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250904 01:55:34.427103 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap complete.
I20250904 01:55:34.431593 5627 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:34.432302 5627 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef804bcecd943d89c7af93e9375b1f3, State: Initialized, Role: FOLLOWER
I20250904 01:55:34.432768 5627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 8.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:34.433347 5631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:34.433537 5631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:34.434212 5627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:34.452912 5645 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:34.453056 5645 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: loading cluster ID for follower catalog manager: success
I20250904 01:55:34.458941 5645 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: acquiring CA information for follower catalog manager: success
I20250904 01:55:34.461668 5645 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:34.467970 5560 master_runner.cc:186] Error getting master registration for 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111),
I20250904 01:55:34.471243 5560 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:34.692035 4030 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:34.694720 5592 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:49097; Asking this server to re-register.
I20250904 01:55:34.695426 4030 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:34.697190 5592 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:34.705497 5650 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:34.705657 5650 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:34.706272 5650 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:34.708704 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:34.713178 5602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 9 candidate_status { last_received { term: 7 index: 49 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:34.713658 5602 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29b8a62cc8d04df781f991da2bd83ccc for term 9 because replica has last-logged OpId of term: 8 index: 50, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 49.
I20250904 01:55:34.714315 5473 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:34.714596 5650 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250904 01:55:34.734076 4168 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:34.734753 4303 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:34.735513 5591 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:36569; Asking this server to re-register.
I20250904 01:55:34.735988 4168 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:34.736085 5592 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:56923; Asking this server to re-register.
I20250904 01:55:34.736631 4303 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:34.737347 5592 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:34.737862 5591 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
I20250904 01:55:34.822373 5631 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:34.822594 5631 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:34.824227 5631 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:34.824807 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:34.824779 5498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 9 candidate_status { last_received { term: 8 index: 50 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc" is_pre_election: true
I20250904 01:55:34.825191 5498 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef804bcecd943d89c7af93e9375b1f3 in term 8.
I20250904 01:55:34.825727 5576 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters:
I20250904 01:55:34.826069 5631 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250904 01:55:34.826195 5631 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:34.826329 5631 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 8 FOLLOWER]: Advancing to term 9
W20250904 01:55:34.826578 5576 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:34.829545 5631 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:34.829975 5631 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 9 election: Requested vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:34.830292 5498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef804bcecd943d89c7af93e9375b1f3" candidate_term: 9 candidate_status { last_received { term: 8 index: 50 } } ignore_live_leader: false dest_uuid: "29b8a62cc8d04df781f991da2bd83ccc"
I20250904 01:55:34.830435 5498 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 8 FOLLOWER]: Advancing to term 9
W20250904 01:55:34.830996 5576 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:34.833482 5498 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef804bcecd943d89c7af93e9375b1f3 in term 9.
I20250904 01:55:34.833823 5576 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:34.834089 5631 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 9 FOLLOWER]: Leader election won for term 9
I20250904 01:55:34.834600 5631 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 9 LEADER]: Becoming Leader. State: Replica: 6ef804bcecd943d89c7af93e9375b1f3, State: Running, Role: LEADER
I20250904 01:55:34.835023 5631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 8.50, Last appended by leader: 50, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:34.837352 5656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef804bcecd943d89c7af93e9375b1f3. Latest consensus state: current_term: 9 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:34.837548 5656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:34.838088 5658 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:34.840386 5658 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:34.840724 5658 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:34.840988 5658 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:34.841250 5658 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:34.841490 5658 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:34.841742 5658 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:34.841974 5658 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:34.842242 5658 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:34.842517 5658 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:34.844192 5658 tablet_loader.cc:96] loaded metadata for tablet 138000c0bebd4169a7d2db4a7f660845 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:34.844729 5658 tablet_loader.cc:96] loaded metadata for tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:34.845203 5658 tablet_loader.cc:96] loaded metadata for tablet 5b42841c537742dab8de6bec2dcbcf29 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:34.845646 5658 tablet_loader.cc:96] loaded metadata for tablet 99c7b2028e1f42e99074eee61163dbd9 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:34.846033 5658 tablet_loader.cc:96] loaded metadata for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:34.846575 5658 tablet_loader.cc:96] loaded metadata for tablet c0d920867c2c426586568745632c810c (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:34.846957 5658 tablet_loader.cc:96] loaded metadata for tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:34.847131 5658 tablet_loader.cc:96] loaded metadata for tablet eea8675339d24616b52d0c53c8f48a6f (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:34.847363 5658 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:34.847532 5658 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:34.848445 5658 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:34.848510 5658 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:34.849640 5658 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:34.850502 5658 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Loaded TSK: 0
I20250904 01:55:34.851138 5658 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:34.851924 5658 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:34.852718 5658 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:34.921875 5591 master_service.cc:948] ReplaceTablet: received request to replace tablet 5742132feece4a58a803d0769ee51d05 from {username='slave'} at 127.0.0.1:41354
I20250904 01:55:34.921936 5590 master_service.cc:948] ReplaceTablet: received request to replace tablet 9f05d628c97e4f57ad4cf4337fe6810f from {username='slave'} at 127.0.0.1:41354
I20250904 01:55:34.921887 5592 master_service.cc:948] ReplaceTablet: received request to replace tablet 04b299dd322a4c35bbed9ec90e9fa3b6 from {username='slave'} at 127.0.0.1:41354
I20250904 01:55:34.922405 5588 master_service.cc:948] ReplaceTablet: received request to replace tablet 30378e1466de4f559ac840b676310486 from {username='slave'} at 127.0.0.1:41354
I20250904 01:55:34.922885 5587 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "default.Table_d723b3eb51ff4772a7331c9338cb1918"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:34.923018 5586 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "default.Table_338af7b0a5ed48918a49777a432224f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:34.923144 5584 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "default.Table_c245a71ea8e74145959886fcfac66f07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:34.923504 5589 master_service.cc:948] ReplaceTablet: received request to replace tablet 138000c0bebd4169a7d2db4a7f660845 from {username='slave'} at 127.0.0.1:41354
I20250904 01:55:34.924024 5588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41354:
table { table_name: "default.Table_9a196bab16bb4604ab9b00d45335f022" } new_table_name: "default.Table_09793f14275347b0aaa65279c0b3bc41" modify_external_catalogs: true
I20250904 01:55:34.924438 5583 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41354:
table { table_name: "default.Table_a524c1ec64554d8eb7687bfa84e4579c" } new_table_name: "default.Table_3a626a942a5e41e9b5d523f0be9b7387" modify_external_catalogs: true
I20250904 01:55:34.924530 5585 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "default.Table_d03c97312adf4523be9539809e8fdb5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:34.924893 5591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41354:
table { table_name: "default.Table_e916036e5adf47c886c74e8088132031" } new_table_name: "default.Table_386e7bb279394e4ba14a48582a9ba8f8" modify_external_catalogs: true
I20250904 01:55:34.925837 5590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "default.Table_782e3736e3434cafa6e2825f14c5900b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:34.927218 5592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41354:
table { table_name: "default.Table_7d5858a140ad4b7eb83927265abad9fe" } modify_external_catalogs: true
I20250904 01:55:34.927456 5592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41354:
table { table_name: "default.Table_7d5858a140ad4b7eb83927265abad9fe" } modify_external_catalogs: true
I20250904 01:55:34.930068 5498 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Refusing update from remote peer 6ef804bcecd943d89c7af93e9375b1f3: Log matching property violated. Preceding OpId in replica: term: 7 index: 49. Preceding OpId from leader: term: 9 index: 52. (index mismatch)
W20250904 01:55:34.930810 5576 consensus_peers.cc:489] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 -> Peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Couldn't send request to peer 7f298ba82a5e49028407388810de4b71. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:55:34.931114 5656 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
F20250904 01:55:34.931813 5644 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fa223a1bdcd google::LogMessage::Fail() at ??:0
@ 0x7fa223a1fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa223a1b7cc google::LogMessage::Flush() at ??:0
@ 0x7fa223a1ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fa239e5e7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 01:55:34.944916 5650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef804bcecd943d89c7af93e9375b1f3. Latest consensus state: current_term: 9 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:34.945122 5650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:34.946995 5659 mvcc.cc:204] Tried to move back new op lower bound from 7196471029464584192 to 7196471000449617920. Current Snapshot: MvccSnapshot[applied={T|T < 7196471029464584192}]
@ 0x7fa239e5d061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fa239e64cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250904 01:55:34.955539 5631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:34.955791 5631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:34.955911 5650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:34.956239 5589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Sending DeleteTablet for 3 replicas of tablet 138000c0bebd4169a7d2db4a7f660845
I20250904 01:55:34.956082 5666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:34.956094 5650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:34.956166 5631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "6ef804bcecd943d89c7af93e9375b1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:34.956372 5666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:34.956508 5631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:34.958086 5589 catalog_manager.cc:6652] ReplaceTablet: tablet 138000c0bebd4169a7d2db4a7f660845 deleted and replaced by tablet 73082cf0147a42edb7f03ba225359a47
@ 0x7fa239e64b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250904 01:55:34.967439 5589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "default.Table_968a17a9054a48dda6494188adb73474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fa2267842b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa229ba86db start_thread at ??:0
@ 0x7fa2228cc71f clone at ??:0
W20250904 01:55:35.146025 5674 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:34.964568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.3.77.3:39819)
0904 01:55:34.981424 (+ 16856us) server_negotiation.cc:207] Beginning negotiation
0904 01:55:34.981429 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0904 01:55:35.145629 (+164200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52442: BlockingRecv error: recv got EOF from 127.0.0.1:52442 (error 108)
Metrics: {"server-negotiator.queue_time_us":16762,"thread_start_us":16848,"threads_started":1}
W20250904 01:55:35.146075 5671 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:34.958375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.3.77.1:37169)
0904 01:55:34.973652 (+ 15277us) server_negotiation.cc:207] Beginning negotiation
0904 01:55:34.973657 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0904 01:55:34.973676 (+ 19us) server_negotiation.cc:408] Connection header received
0904 01:55:34.973742 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0904 01:55:34.973748 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0904 01:55:34.973820 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0904 01:55:34.977094 (+ 3274us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0904 01:55:34.977438 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0904 01:55:34.979066 (+ 1628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0904 01:55:35.145771 (+166705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39652: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":15162,"thread_start_us":17589,"threads_started":1}
W20250904 01:55:35.146085 5672 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 01:55:34.958668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.3.77.2:40911)
0904 01:55:34.981652 (+ 22984us) server_negotiation.cc:207] Beginning negotiation
0904 01:55:34.981659 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0904 01:55:34.981676 (+ 17us) server_negotiation.cc:408] Connection header received
0904 01:55:34.981732 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0904 01:55:34.981738 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0904 01:55:34.981805 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0904 01:55:34.981895 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0904 01:55:35.145732 (+163837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50290: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":22861,"thread_start_us":22750,"threads_started":1}
W20250904 01:55:35.146430 4309 connection.cc:537] client connection to 127.3.77.61:36307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:35.146978 4309 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.61:36307)
W20250904 01:55:35.147451 4309 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.61:36307)
W20250904 01:55:35.147610 4309 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.61:36307)
W20250904 01:55:35.147935 4309 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.61:36307)
I20250904 01:55:35.476404 5684 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:35.476608 5684 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:35.477339 5684 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:35.479040 5473 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:35.479254 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:35.479367 5472 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:35.479621 5684 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250904 01:55:36.029682 5689 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:36.029829 5689 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:36.030431 5689 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:36.032042 5473 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:36.032301 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:36.032406 5472 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:36.032622 5689 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250904 01:55:36.255885 5457 master_runner.cc:186] Error getting master registration for 127.3.77.61:36307: Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111),
I20250904 01:55:36.256036 5457 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:36.718942 5694 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:36.719090 5694 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:36.719591 5694 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:36.720955 5473 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307): Network error: Client connection negotiation failed: client connection to 127.3.77.61:36307: connect: Connection refused (error 111)
W20250904 01:55:36.721122 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:36.721238 5472 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc; no voters: 6ef804bcecd943d89c7af93e9375b1f3, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:36.721433 5694 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250904 01:55:37.053136 3380 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
/tmp/dist-test-taskHkVPgV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--webserver_interface=127.3.77.61
--webserver_port=46249
--builtin_ntp_servers=127.3.77.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 01:55:37.344992 5697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:55:37.345228 5697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:55:37.345299 5697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:55:37.352198 5697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:55:37.352262 5697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:55:37.352308 5697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:55:37.352340 5697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:55:37.360570 5697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.77.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:46285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.77.61:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.77.61
--webserver_port=46249
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:23 UTC on 24a791456cd2
build id 7888
ASAN enabled
I20250904 01:55:37.361829 5697 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:55:37.363365 5697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:55:37.370986 5704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:37.371317 5705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:55:37.371665 5707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:55:37.372519 5697 server_base.cc:1047] running on GCE node
I20250904 01:55:37.373075 5697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:55:37.374192 5697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:55:37.375406 5697 hybrid_clock.cc:648] HybridClock initialized: now 1756950937375373 us; error 28 us; skew 500 ppm
I20250904 01:55:37.375675 5697 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:55:37.377562 5697 webserver.cc:480] Webserver started at http://127.3.77.61:46249/ using document root <none> and password file <none>
I20250904 01:55:37.378151 5697 fs_manager.cc:362] Metadata directory not provided
I20250904 01:55:37.378252 5697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:55:37.382669 5697 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:55:37.385064 5713 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:55:37.386189 5697 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250904 01:55:37.386354 5697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data,/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
uuid: "6ef804bcecd943d89c7af93e9375b1f3"
format_stamp: "Formatted at 2025-09-04 01:55:06 on dist-test-slave-ch6q"
I20250904 01:55:37.386992 5697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:55:37.407917 5697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:55:37.408594 5697 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:55:37.408921 5697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:55:37.426605 5697 rpc_server.cc:307] RPC server started. Bound to: 127.3.77.61:36307
I20250904 01:55:37.426641 5764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.77.61:36307 every 8 connection(s)
I20250904 01:55:37.427987 5697 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/master-1/data/info.pb
I20250904 01:55:37.428987 3380 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskHkVPgV/build/asan/bin/kudu as pid 5697
I20250904 01:55:37.436051 5765 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:55:37.440680 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap starting.
I20250904 01:55:37.452585 5765 log.cc:826] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Log is configured to *not* fsync() on all Append() calls
I20250904 01:55:37.488291 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=33 ignored=0} mutations{seen=40 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:55:37.488919 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: Bootstrap complete.
I20250904 01:55:37.493510 5765 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:37.494102 5765 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef804bcecd943d89c7af93e9375b1f3, State: Initialized, Role: FOLLOWER
I20250904 01:55:37.494678 5765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 9.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:37.495337 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:37.495528 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:37.496562 5765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:55:37.516462 5784 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:37.516599 5784 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: loading cluster ID for follower catalog manager: success
I20250904 01:55:37.520458 5784 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: acquiring CA information for follower catalog manager: success
I20250904 01:55:37.531306 5784 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:55:37.545392 5697 master_runner.cc:186] Error getting master registration for 127.3.77.60:38761: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111),
I20250904 01:55:37.548885 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:37.708809 4030 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:37.710966 5730 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" instance_seqno: 1756950907666591) as {username='slave'} at 127.3.77.1:36787; Asking this server to re-register.
I20250904 01:55:37.711541 4030 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:37.712893 5730 ts_manager.cc:194] Registered new tserver with Master: 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:37.741925 5789 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:37.742069 5789 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:37.743014 5789 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:37.753033 4303 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:37.754848 5730 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7d96d300478849e3b11402fbd34fef15" instance_seqno: 1756950908505557) as {username='slave'} at 127.3.77.3:32777; Asking this server to re-register.
I20250904 01:55:37.755623 4303 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:37.757470 5730 ts_manager.cc:194] Registered new tserver with Master: 7d96d300478849e3b11402fbd34fef15 (127.3.77.3:39819)
W20250904 01:55:37.758587 5472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250904 01:55:37.760032 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:37.762480 5740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 10 candidate_status { last_received { term: 9 index: 53 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3" is_pre_election: true
I20250904 01:55:37.763056 5740 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 9.
I20250904 01:55:37.763623 5473 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:37.764071 5789 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250904 01:55:37.764183 5789 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6ef804bcecd943d89c7af93e9375b1f3)
I20250904 01:55:37.764268 5789 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 9 FOLLOWER]: Advancing to term 10
I20250904 01:55:37.765388 4168 heartbeater.cc:344] Connected to a master server at 127.3.77.61:36307
I20250904 01:55:37.766683 5789 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:37.766731 5730 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" instance_seqno: 1756950908103365) as {username='slave'} at 127.3.77.2:44837; Asking this server to re-register.
I20250904 01:55:37.767292 5789 leader_election.cc:290] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 election: Requested vote from peers 6ef804bcecd943d89c7af93e9375b1f3 (127.3.77.61:36307), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
I20250904 01:55:37.767333 4168 heartbeater.cc:461] Registering TS with master...
I20250904 01:55:37.767652 5740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29b8a62cc8d04df781f991da2bd83ccc" candidate_term: 10 candidate_status { last_received { term: 9 index: 53 } } ignore_live_leader: false dest_uuid: "6ef804bcecd943d89c7af93e9375b1f3"
I20250904 01:55:37.767823 5740 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 9 FOLLOWER]: Advancing to term 10
W20250904 01:55:37.768792 5472 leader_election.cc:336] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:37.773254 5740 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b8a62cc8d04df781f991da2bd83ccc in term 10.
I20250904 01:55:37.773639 5473 leader_election.cc:304] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b8a62cc8d04df781f991da2bd83ccc, 6ef804bcecd943d89c7af93e9375b1f3; no voters: 7f298ba82a5e49028407388810de4b71
I20250904 01:55:37.773965 5789 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 10 FOLLOWER]: Leader election won for term 10
I20250904 01:55:37.774510 5730 ts_manager.cc:194] Registered new tserver with Master: 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911)
I20250904 01:55:37.774649 5789 raft_consensus.cc:695] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [term 10 LEADER]: Becoming Leader. State: Replica: 29b8a62cc8d04df781f991da2bd83ccc, State: Running, Role: LEADER
I20250904 01:55:37.775084 5789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 9.53, Last appended by leader: 53, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:37.776515 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 10 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:37.776710 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:37.777407 5796 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:55:37.779667 5796 catalog_manager.cc:671] Loaded metadata for table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda]
I20250904 01:55:37.780061 5796 catalog_manager.cc:671] Loaded metadata for table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57]
I20250904 01:55:37.780364 5796 catalog_manager.cc:671] Loaded metadata for table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34]
I20250904 01:55:37.780663 5796 catalog_manager.cc:671] Loaded metadata for table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512]
I20250904 01:55:37.780951 5796 catalog_manager.cc:671] Loaded metadata for table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]
I20250904 01:55:37.781275 5796 catalog_manager.cc:671] Loaded metadata for table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]
I20250904 01:55:37.781564 5796 catalog_manager.cc:671] Loaded metadata for table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da]
I20250904 01:55:37.781867 5796 catalog_manager.cc:671] Loaded metadata for table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19]
I20250904 01:55:37.782182 5796 catalog_manager.cc:671] Loaded metadata for table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b]
I20250904 01:55:37.784217 5796 tablet_loader.cc:96] loaded metadata for tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe])
I20250904 01:55:37.784787 5796 tablet_loader.cc:96] loaded metadata for tablet 5b42841c537742dab8de6bec2dcbcf29 (table default.table_5a0ac2833a3d4a2b98f538f7416ca2d1 [id=d80dffee0777462c92f9d1f25fc9ad19])
I20250904 01:55:37.785198 5796 tablet_loader.cc:96] loaded metadata for tablet 73082cf0147a42edb7f03ba225359a47 (table default.table_d150897ff208409984d49a17e902c6c2 [id=16449e4c6e234d69b3d2926853ee1cda])
I20250904 01:55:37.785579 5796 tablet_loader.cc:96] loaded metadata for tablet 99c7b2028e1f42e99074eee61163dbd9 (table default.table_9a196bab16bb4604ab9b00d45335f022 [id=e4d470eb83d64f2284134d253af56c6b])
I20250904 01:55:37.786023 5796 tablet_loader.cc:96] loaded metadata for tablet ac85bb6dc61f4d0cbe26f94b4e5c0f96 (table default.table_e916036e5adf47c886c74e8088132031 [id=16a11a5e690e4cbb836c8553b6834a57])
I20250904 01:55:37.786616 5796 tablet_loader.cc:96] loaded metadata for tablet c0d920867c2c426586568745632c810c (table default.table_9c770f9d8cf94c06a272116f40e60c9b [id=6b3e7d9283484d05afe431627ccc2b34])
I20250904 01:55:37.787055 5796 tablet_loader.cc:96] loaded metadata for tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693])
I20250904 01:55:37.787340 5796 tablet_loader.cc:96] loaded metadata for tablet eea8675339d24616b52d0c53c8f48a6f (table default.table_a89876a7c4d946f09bfa330cab1f34f2 [id=c5047cd5db7c47348502bcad97d7d2da])
I20250904 01:55:37.787633 5796 tablet_loader.cc:96] loaded metadata for tablet fd66dd528eb947ceb75bcbd518eb2ee4 (table default.table_18be6a726bc64f8c8efad17c700b66aa [id=7bc49ad0e845460ead08c387215ff512])
I20250904 01:55:37.787851 5796 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:55:37.788763 5796 catalog_manager.cc:1261] Loaded cluster ID: 3db05ef0f66f409499605231169f46b8
I20250904 01:55:37.788834 5796 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:55:37.790216 5796 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:55:37.791043 5796 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Loaded TSK: 0
I20250904 01:55:37.791770 5796 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:55:37.792696 5796 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:55:37.793499 5796 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:55:37.810740 5487 master_service.cc:948] ReplaceTablet: received request to replace tablet ba52aec4f85c4723b600ab9271beb725 from {username='slave'} at 127.0.0.1:48710
I20250904 01:55:37.811486 5488 master_service.cc:948] ReplaceTablet: received request to replace tablet 3fa7e488232a450dbe28c7d2252a7a1c from {username='slave'} at 127.0.0.1:48710
I20250904 01:55:37.812045 5481 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48710:
name: "default.Table_39d410e70d9349abad85c4dccbad54bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:37.813174 5482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_5a0ac2833a3d4a2b98f538f7416ca2d1" } new_table_name: "default.Table_ae25e811a20d449d8d8c57cca63f758b" modify_external_catalogs: true
I20250904 01:55:37.813725 5483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_18be6a726bc64f8c8efad17c700b66aa" } new_table_name: "default.Table_db0617ec7a994e45b7e7ed190308ddbe" modify_external_catalogs: true
I20250904 01:55:37.813803 5480 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:37.814118 5480 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_d150897ff208409984d49a17e902c6c2" } modify_external_catalogs: true
I20250904 01:55:37.814313 5484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48710:
name: "default.Table_bc5b38d3266841d6b4a43d4983cff6b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:37.814603 5479 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:37.814848 5479 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_a89876a7c4d946f09bfa330cab1f34f2" } modify_external_catalogs: true
I20250904 01:55:37.815009 5485 master_service.cc:948] ReplaceTablet: received request to replace tablet ce814e705d2a4a5cafb5670424c008f0 from {username='slave'} at 127.0.0.1:48710
I20250904 01:55:37.816234 5486 master_service.cc:948] ReplaceTablet: received request to replace tablet 9f05d628c97e4f57ad4cf4337fe6810f from {username='slave'} at 127.0.0.1:48710
I20250904 01:55:37.817461 5486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48710:
name: "default.Table_bce07055a7064408b762f1534d59a90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:37.818059 5487 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b" } modify_external_catalogs: true
I20250904 01:55:37.818300 5487 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48710:
table { table_name: "default.Table_9c770f9d8cf94c06a272116f40e60c9b" } modify_external_catalogs: true
I20250904 01:55:37.819777 5740 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Refusing update from remote peer 29b8a62cc8d04df781f991da2bd83ccc: Log matching property violated. Preceding OpId in replica: term: 9 index: 53. Preceding OpId from leader: term: 10 index: 55. (index mismatch)
W20250904 01:55:37.821517 5472 consensus_peers.cc:489] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc -> Peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Couldn't send request to peer 7f298ba82a5e49028407388810de4b71. Status: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:55:37.821818 5794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
F20250904 01:55:37.823127 5540 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_9a196bab16bb4604ab9b00d45335f022, message={"server":"","servicePrincipal":"","db":"default","table":"table_9a196bab16bb4604ab9b00d45335f022","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9a196bab16bb4604ab9b00d45335f022\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_9a196bab16bb4604ab9b00d45335f022\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9a196bab16bb4604ab9b00d45335f022\",\"kudu.table_id\":\"e4d470eb83d64f2284134d253af56c6b\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756950908, eventType=CREATE_TABLE, dbName=default, tableName=table_ac1709b4bdce4074b1fac1b503f0d518, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac1709b4bdce4074b1fac1b503f0d518","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac1709b4bdce4074b1fac1b503f0d518\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756950908},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/warehouse/table_ac1709b4bdce4074b1fac1b503f0d518\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac1709b4bdce4074b1fac1b503f0d518\",\"kudu.table_id\":\"ef1c5970c2c54d698c3fdbda5e5245c5\",\"transient_lastDdlTime\":\"1756950908\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3db05ef0f66f409499605231169f46b8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756950908,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f646d495dcd google::LogMessage::Fail() at ??:0
@ 0x7f646d499b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f646d4957cc google::LogMessage::Flush() at ??:0
@ 0x7f646d495a11 google::LogMessage::~LogMessage() at ??:0
I20250904 01:55:37.833074 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8a62cc8d04df781f991da2bd83ccc. Latest consensus state: current_term: 10 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:37.833323 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:37.834841 5797 mvcc.cc:204] Tried to move back new op lower bound from 7196471041296461824 to 7196471041129635840. Current Snapshot: MvccSnapshot[applied={T|T < 7196471041296461824}]
@ 0x7f64838d87b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 01:55:37.844089 5488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Sending DeleteTablet for 3 replicas of tablet 3fa7e488232a450dbe28c7d2252a7a1c
I20250904 01:55:37.846163 5488 catalog_manager.cc:6652] ReplaceTablet: tablet 3fa7e488232a450dbe28c7d2252a7a1c deleted and replaced by tablet b41d879e5db948c2973efcf779ba7925
I20250904 01:55:37.856217 5485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc: Sending DeleteTablet for 3 replicas of tablet ce814e705d2a4a5cafb5670424c008f0
I20250904 01:55:37.856683 5485 catalog_manager.cc:6652] ReplaceTablet: tablet ce814e705d2a4a5cafb5670424c008f0 deleted and replaced by tablet 8ca1f7ac965542069e8068348e8390dc
@ 0x7f64838d7061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f64838decdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250904 01:55:37.869799 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:37.870005 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:55:37.871181 5488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48710:
name: "default.Table_d723b3eb51ff4772a7331c9338cb1918"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:37.871716 5485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48710:
name: "default.Table_338af7b0a5ed48918a49777a432224f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 01:55:37.872112 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "29b8a62cc8d04df781f991da2bd83ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } } }
I20250904 01:55:37.872315 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8a62cc8d04df781f991da2bd83ccc [sys.catalog]: This master's current role is: LEADER
I20250904 01:55:37.876255 4098 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fa7e488232a450dbe28c7d2252a7a1c with delete_type TABLET_DATA_DELETED (replaced by tablet b41d879e5db948c2973efcf779ba7925) from {username='slave'} at 127.0.0.1:50292
I20250904 01:55:37.876574 4097 tablet_service.cc:1515] Processing DeleteTablet for tablet ce814e705d2a4a5cafb5670424c008f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ca1f7ac965542069e8068348e8390dc) from {username='slave'} at 127.0.0.1:50292
I20250904 01:55:37.880100 4235 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fa7e488232a450dbe28c7d2252a7a1c with delete_type TABLET_DATA_DELETED (replaced by tablet b41d879e5db948c2973efcf779ba7925) from {username='slave'} at 127.0.0.1:52456
I20250904 01:55:37.880272 4236 tablet_service.cc:1515] Processing DeleteTablet for tablet ce814e705d2a4a5cafb5670424c008f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ca1f7ac965542069e8068348e8390dc) from {username='slave'} at 127.0.0.1:52456
I20250904 01:55:37.880750 5813 tablet_replica.cc:331] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: stopping tablet replica
I20250904 01:55:37.880995 5813 raft_consensus.cc:2241] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Raft consensus shutting down.
I20250904 01:55:37.881412 5813 raft_consensus.cc:2270] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:37.882071 5813 ts_tablet_manager.cc:1905] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:37.883217 3963 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fa7e488232a450dbe28c7d2252a7a1c with delete_type TABLET_DATA_DELETED (replaced by tablet b41d879e5db948c2973efcf779ba7925) from {username='slave'} at 127.0.0.1:39664
I20250904 01:55:37.883518 3962 tablet_service.cc:1515] Processing DeleteTablet for tablet ce814e705d2a4a5cafb5670424c008f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ca1f7ac965542069e8068348e8390dc) from {username='slave'} at 127.0.0.1:39664
@ 0x7f64838deb59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250904 01:55:37.894670 5815 tablet_replica.cc:331] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: stopping tablet replica
I20250904 01:55:37.894861 5815 raft_consensus.cc:2241] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:37.895094 5815 raft_consensus.cc:2270] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:37.895637 5815 ts_tablet_manager.cc:1905] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:37.896309 5813 ts_tablet_manager.cc:1918] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:37.896415 5813 log.cc:1199] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/wals/3fa7e488232a450dbe28c7d2252a7a1c
I20250904 01:55:37.896801 5813 ts_tablet_manager.cc:1939] T 3fa7e488232a450dbe28c7d2252a7a1c P 5b23e87ab60248f391fb17326f5e3934: Deleting consensus metadata
I20250904 01:55:37.897652 5813 tablet_replica.cc:331] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: stopping tablet replica
I20250904 01:55:37.897833 5813 raft_consensus.cc:2241] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 1 LEADER]: Raft consensus shutting down.
I20250904 01:55:37.897810 5474 catalog_manager.cc:4928] TS 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911): tablet 3fa7e488232a450dbe28c7d2252a7a1c (table default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]) successfully deleted
I20250904 01:55:37.898169 5813 raft_consensus.cc:2270] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:37.898816 5813 ts_tablet_manager.cc:1905] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:37.900548 4097 tablet_service.cc:1468] Processing CreateTablet for tablet 8ca1f7ac965542069e8068348e8390dc (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:37.900890 4097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca1f7ac965542069e8068348e8390dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:37.901082 4236 tablet_service.cc:1468] Processing CreateTablet for tablet b41d879e5db948c2973efcf779ba7925 (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:37.901436 4236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b41d879e5db948c2973efcf779ba7925. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:37.902557 4234 tablet_service.cc:1468] Processing CreateTablet for tablet 8ca1f7ac965542069e8068348e8390dc (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:37.902930 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca1f7ac965542069e8068348e8390dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:37.904616 5813 ts_tablet_manager.cc:1918] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:37.904726 5813 log.cc:1199] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-1/wal/wals/ce814e705d2a4a5cafb5670424c008f0
I20250904 01:55:37.905092 5813 ts_tablet_manager.cc:1939] T ce814e705d2a4a5cafb5670424c008f0 P 5b23e87ab60248f391fb17326f5e3934: Deleting consensus metadata
I20250904 01:55:37.905894 5474 catalog_manager.cc:4928] TS 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911): tablet ce814e705d2a4a5cafb5670424c008f0 (table default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]) successfully deleted
I20250904 01:55:37.906852 3962 tablet_service.cc:1468] Processing CreateTablet for tablet b41d879e5db948c2973efcf779ba7925 (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:37.907210 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b41d879e5db948c2973efcf779ba7925. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:37.908733 3961 tablet_service.cc:1468] Processing CreateTablet for tablet 8ca1f7ac965542069e8068348e8390dc (DEFAULT_TABLE table=default.table_7d5858a140ad4b7eb83927265abad9fe [id=abe669ac909041a1929e0c8590110693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:37.909164 3961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca1f7ac965542069e8068348e8390dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:37.899905 4098 tablet_service.cc:1468] Processing CreateTablet for tablet b41d879e5db948c2973efcf779ba7925 (DEFAULT_TABLE table=default.table_a524c1ec64554d8eb7687bfa84e4579c [id=8fb7faa10e1143d6a57c150762463cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:55:37.917227 4098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b41d879e5db948c2973efcf779ba7925. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:55:37.917765 5818 tablet_bootstrap.cc:492] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
@ 0x7f64701fe2b9 kudu::Thread::SuperviseThread() at ??:0
I20250904 01:55:37.919847 5818 tablet_bootstrap.cc:654] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
@ 0x7f64736226db start_thread at ??:0
@ 0x7f646c34671f clone at ??:0
I20250904 01:55:37.922399 5819 tablet_bootstrap.cc:492] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:37.923854 5819 tablet_bootstrap.cc:654] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:37.924793 5821 tablet_bootstrap.cc:492] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:37.927592 5821 tablet_bootstrap.cc:654] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:37.927774 5814 tablet_replica.cc:331] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: stopping tablet replica
I20250904 01:55:37.927963 5814 raft_consensus.cc:2241] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:37.928181 5814 raft_consensus.cc:2270] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:37.928784 5814 ts_tablet_manager.cc:1905] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:37.931339 5818 tablet_bootstrap.cc:492] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:37.931471 5818 ts_tablet_manager.cc:1397] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20250904 01:55:37.932195 5818 raft_consensus.cc:357] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.932384 5818 raft_consensus.cc:383] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:37.932456 5818 raft_consensus.cc:738] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:37.932670 5818 consensus_queue.cc:260] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.934490 5815 ts_tablet_manager.cc:1918] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:37.934608 5815 log.cc:1199] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/wals/3fa7e488232a450dbe28c7d2252a7a1c
I20250904 01:55:37.935065 5815 ts_tablet_manager.cc:1939] T 3fa7e488232a450dbe28c7d2252a7a1c P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting consensus metadata
I20250904 01:55:37.937359 5818 ts_tablet_manager.cc:1428] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250904 01:55:37.940316 5818 tablet_bootstrap.cc:492] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15: Bootstrap starting.
I20250904 01:55:37.941777 5818 tablet_bootstrap.cc:654] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:37.941809 5814 ts_tablet_manager.cc:1918] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:37.941895 5814 log.cc:1199] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/wals/3fa7e488232a450dbe28c7d2252a7a1c
I20250904 01:55:37.942363 5814 ts_tablet_manager.cc:1939] T 3fa7e488232a450dbe28c7d2252a7a1c P 7d96d300478849e3b11402fbd34fef15: Deleting consensus metadata
I20250904 01:55:37.943202 5814 tablet_replica.cc:331] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: stopping tablet replica
I20250904 01:55:37.943390 5814 raft_consensus.cc:2241] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:37.943536 5814 raft_consensus.cc:2270] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:37.943781 5821 tablet_bootstrap.cc:492] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:37.943897 5821 ts_tablet_manager.cc:1397] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250904 01:55:37.944016 5814 ts_tablet_manager.cc:1905] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:37.944599 5821 raft_consensus.cc:357] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.944762 5821 raft_consensus.cc:383] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:37.944835 5821 raft_consensus.cc:738] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:37.945073 5821 consensus_queue.cc:260] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.946394 5815 tablet_replica.cc:331] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: stopping tablet replica
I20250904 01:55:37.946619 5815 raft_consensus.cc:2241] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 01:55:37.946792 5815 raft_consensus.cc:2270] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 01:55:37.947340 5815 ts_tablet_manager.cc:1905] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 01:55:37.948316 5821 ts_tablet_manager.cc:1428] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250904 01:55:37.951351 5821 tablet_bootstrap.cc:492] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934: Bootstrap starting.
I20250904 01:55:37.952787 5821 tablet_bootstrap.cc:654] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934: Neither blocks nor log segments found. Creating new log.
I20250904 01:55:37.953151 5815 ts_tablet_manager.cc:1918] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:37.953284 5815 log.cc:1199] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-0/wal/wals/ce814e705d2a4a5cafb5670424c008f0
I20250904 01:55:37.953743 5815 ts_tablet_manager.cc:1939] T ce814e705d2a4a5cafb5670424c008f0 P 2ca66c5fca0c4305a2857a0074a8f8cf: Deleting consensus metadata
I20250904 01:55:37.955966 5818 tablet_bootstrap.cc:492] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15: No bootstrap required, opened a new log
I20250904 01:55:37.956153 5818 ts_tablet_manager.cc:1397] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250904 01:55:37.956472 5814 ts_tablet_manager.cc:1918] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 01:55:37.956625 5814 log.cc:1199] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: Deleting WAL directory at /tmp/dist-test-taskHkVPgV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756950892568603-3380-0/minicluster-data/ts-2/wal/wals/ce814e705d2a4a5cafb5670424c008f0
I20250904 01:55:37.957084 5814 ts_tablet_manager.cc:1939] T ce814e705d2a4a5cafb5670424c008f0 P 7d96d300478849e3b11402fbd34fef15: Deleting consensus metadata
I20250904 01:55:37.957008 5818 raft_consensus.cc:357] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.957234 5818 raft_consensus.cc:383] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:37.957337 5818 raft_consensus.cc:738] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Initialized, Role: FOLLOWER
I20250904 01:55:37.958153 5819 tablet_bootstrap.cc:492] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:37.958334 5819 ts_tablet_manager.cc:1397] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.003s
I20250904 01:55:37.959278 5819 raft_consensus.cc:357] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.959509 5819 raft_consensus.cc:383] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:37.959607 5819 raft_consensus.cc:738] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:37.959846 5819 consensus_queue.cc:260] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.960415 5819 ts_tablet_manager.cc:1428] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250904 01:55:37.960641 5818 consensus_queue.cc:260] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.963418 5702 raft_consensus.cc:491] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:37.963533 5702 raft_consensus.cc:513] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.963641 5819 tablet_bootstrap.cc:492] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf: Bootstrap starting.
I20250904 01:55:37.964385 5702 leader_election.cc:290] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:37.964841 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b41d879e5db948c2973efcf779ba7925" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:37.965533 3982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b41d879e5db948c2973efcf779ba7925" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" is_pre_election: true
I20250904 01:55:37.965723 5818 ts_tablet_manager.cc:1428] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250904 01:55:37.965709 3982 raft_consensus.cc:2466] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 0.
I20250904 01:55:37.965761 5819 tablet_bootstrap.cc:654] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf: Neither blocks nor log segments found. Creating new log.
W20250904 01:55:37.966717 4191 leader_election.cc:343] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911): Illegal state: must be running to vote when last-logged opid is not known
I20250904 01:55:37.966992 4190 leader_election.cc:304] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters: 5b23e87ab60248f391fb17326f5e3934
I20250904 01:55:37.967408 5702 raft_consensus.cc:2802] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:37.967514 5702 raft_consensus.cc:491] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:37.967589 5702 raft_consensus.cc:3058] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:37.968698 5819 tablet_bootstrap.cc:492] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf: No bootstrap required, opened a new log
I20250904 01:55:37.968837 5819 ts_tablet_manager.cc:1397] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 01:55:37.969007 5821 tablet_bootstrap.cc:492] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934: No bootstrap required, opened a new log
I20250904 01:55:37.969151 5821 ts_tablet_manager.cc:1397] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20250904 01:55:37.969583 5819 raft_consensus.cc:357] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.969736 5819 raft_consensus.cc:383] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:37.969815 5819 raft_consensus.cc:738] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca66c5fca0c4305a2857a0074a8f8cf, State: Initialized, Role: FOLLOWER
I20250904 01:55:37.969797 5821 raft_consensus.cc:357] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.969978 5821 raft_consensus.cc:383] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:55:37.970077 5821 raft_consensus.cc:738] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23e87ab60248f391fb17326f5e3934, State: Initialized, Role: FOLLOWER
I20250904 01:55:37.970073 5702 raft_consensus.cc:513] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.970046 5819 consensus_queue.cc:260] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.970332 5821 consensus_queue.cc:260] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:37.970572 5702 leader_election.cc:290] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:37.971928 5819 ts_tablet_manager.cc:1428] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 01:55:37.972398 3982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b41d879e5db948c2973efcf779ba7925" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
I20250904 01:55:37.972558 3982 raft_consensus.cc:3058] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:37.972558 5821 ts_tablet_manager.cc:1428] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 01:55:37.974288 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b41d879e5db948c2973efcf779ba7925" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:37.974443 4120 raft_consensus.cc:3058] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:37.975375 3982 raft_consensus.cc:2466] T b41d879e5db948c2973efcf779ba7925 P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 1.
I20250904 01:55:37.975730 4190 leader_election.cc:304] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:37.976027 5702 raft_consensus.cc:2802] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:37.977053 4120 raft_consensus.cc:2466] T b41d879e5db948c2973efcf779ba7925 P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 1.
I20250904 01:55:37.978367 5702 raft_consensus.cc:695] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [term 1 LEADER]: Becoming Leader. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Running, Role: LEADER
I20250904 01:55:37.978649 5702 consensus_queue.cc:237] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
W20250904 01:55:38.102556 3918 connection.cc:537] server connection from 127.0.0.1:39664 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:38.102557 4190 connection.cc:537] server connection from 127.0.0.1:52456 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:38.102568 4055 connection.cc:537] server connection from 127.0.0.1:50292 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:38.102954 3915 connection.cc:537] client connection to 127.3.77.62:34273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:38.104208 4308 connection.cc:537] client connection to 127.3.77.62:34273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:55:38.104857 4308 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.77.62:34273)
W20250904 01:55:38.105358 4308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:38.105621 4308 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.77.62:34273)
W20250904 01:55:38.105926 4308 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.77.62:34273)
I20250904 01:55:38.211774 5702 raft_consensus.cc:491] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:55:38.211934 5702 raft_consensus.cc:513] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:38.212415 5702 leader_election.cc:290] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:38.212775 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ca1f7ac965542069e8068348e8390dc" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934" is_pre_election: true
I20250904 01:55:38.212808 3982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ca1f7ac965542069e8068348e8390dc" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" is_pre_election: true
I20250904 01:55:38.212967 4120 raft_consensus.cc:2466] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 0.
I20250904 01:55:38.212993 3982 raft_consensus.cc:2466] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 0.
I20250904 01:55:38.213304 4191 leader_election.cc:304] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:38.213568 5702 raft_consensus.cc:2802] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:55:38.213668 5702 raft_consensus.cc:491] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:55:38.213768 5702 raft_consensus.cc:3058] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:38.215859 5702 raft_consensus.cc:513] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:38.216377 5702 leader_election.cc:290] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 election: Requested vote from peers 5b23e87ab60248f391fb17326f5e3934 (127.3.77.2:40911), 2ca66c5fca0c4305a2857a0074a8f8cf (127.3.77.1:37169)
I20250904 01:55:38.216686 4120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ca1f7ac965542069e8068348e8390dc" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23e87ab60248f391fb17326f5e3934"
I20250904 01:55:38.216764 3982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ca1f7ac965542069e8068348e8390dc" candidate_uuid: "7d96d300478849e3b11402fbd34fef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf"
I20250904 01:55:38.216866 4120 raft_consensus.cc:3058] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:38.216966 3982 raft_consensus.cc:3058] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:55:38.219522 3982 raft_consensus.cc:2466] T 8ca1f7ac965542069e8068348e8390dc P 2ca66c5fca0c4305a2857a0074a8f8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 1.
I20250904 01:55:38.219522 4120 raft_consensus.cc:2466] T 8ca1f7ac965542069e8068348e8390dc P 5b23e87ab60248f391fb17326f5e3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d96d300478849e3b11402fbd34fef15 in term 1.
I20250904 01:55:38.219962 4190 leader_election.cc:304] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ca66c5fca0c4305a2857a0074a8f8cf, 5b23e87ab60248f391fb17326f5e3934, 7d96d300478849e3b11402fbd34fef15; no voters:
I20250904 01:55:38.220186 5702 raft_consensus.cc:2802] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:55:38.220320 5702 raft_consensus.cc:695] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [term 1 LEADER]: Becoming Leader. State: Replica: 7d96d300478849e3b11402fbd34fef15, State: Running, Role: LEADER
I20250904 01:55:38.220557 5702 consensus_queue.cc:237] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96d300478849e3b11402fbd34fef15" member_type: VOTER last_known_addr { host: "127.3.77.3" port: 39819 } } peers { permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 } } peers { permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 } }
I20250904 01:55:38.398865 5836 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:38.399046 5836 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:38.400604 5836 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:38.401495 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:38.402616 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:38.402824 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:38.402920 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:38.403199 5836 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:38.961411 5842 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:38.961562 5842 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:38.962239 5842 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:38.964021 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:38.964181 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:38.964303 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:38.964515 5842 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:39.004630 5846 consensus_queue.cc:1035] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:39.011379 5852 consensus_queue.cc:1035] T b41d879e5db948c2973efcf779ba7925 P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:39.049789 5852 consensus_queue.cc:1035] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23e87ab60248f391fb17326f5e3934" member_type: VOTER last_known_addr { host: "127.3.77.2" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:39.054039 5848 consensus_queue.cc:1035] T 8ca1f7ac965542069e8068348e8390dc P 7d96d300478849e3b11402fbd34fef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca66c5fca0c4305a2857a0074a8f8cf" member_type: VOTER last_known_addr { host: "127.3.77.1" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:55:39.830955 5866 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:39.831116 5866 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:39.831629 5866 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:39.833637 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:39.833851 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:39.833957 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:39.834265 5866 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:39.972652 3380 master-stress-test.cc:577] Killing test thread 1/25
I20250904 01:55:40.366614 5877 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:40.366753 5877 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:40.367285 5877 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:40.369903 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:40.370071 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:40.370174 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:40.370417 5877 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:41.761739 5895 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:41.761891 5895 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:41.762415 5895 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:41.763953 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:41.764122 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:41.764202 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:41.764429 5895 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:42.549930 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:55:42.550076 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:42.894263 5909 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:42.894418 5909 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:42.894995 5909 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:42.898190 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:42.898393 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:42.898473 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:42.898660 5909 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:46.554543 5948 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:46.554697 5948 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:46.555153 5948 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:46.556058 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20250904 01:55:46.559257 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:46.559427 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:46.559531 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:46.559773 5948 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:47.551352 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:55:47.551522 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:49.455193 5984 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:49.455386 5984 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:49.455921 5984 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:49.457990 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:49.458194 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:49.458333 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:49.458536 5984 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:52.557894 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:55:52.558075 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:55:54.510412 6042 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:55:54.510637 6042 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:55:54.511428 6042 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:55:54.512456 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 01:55:54.523352 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:55:54.523555 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:55:54.523643 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:55:54.523873 6042 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:55:57.559264 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:55:57.559424 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:02.560570 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:02.560724 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:07.561991 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:07.562157 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:56:09.175498 4166 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (115 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:56:09.175732 4029 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (66 consecutive failures): Network error: Failed to ping master at 127.3.77.62:34273: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:09.314775 4301 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (116 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:11.262548 6230 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:56:11.262745 6230 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:56:11.263499 6230 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:56:11.264619 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:56:11.265774 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:11.265947 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:11.266021 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:56:11.266238 6230 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:56:12.563573 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:12.563743 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:17.564945 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:17.565104 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:22.566205 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:22.566393 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:22.819972 6356 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:56:22.820150 6356 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:56:22.820822 6356 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:56:22.821486 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:56:22.822728 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:22.822901 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:22.822975 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:56:22.823177 6356 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:56:27.431830 6409 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:56:27.432037 6409 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:56:27.432756 6409 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:56:27.435021 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:27.435212 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:27.435321 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:56:27.435552 6409 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:56:27.567282 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:27.567466 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:32.568849 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:32.569031 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:37.570276 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:37.570451 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:39.057590 6538 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:56:39.057787 6538 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:56:39.058399 6538 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:56:39.059465 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 01:56:39.063382 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:39.063558 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:39.063644 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:56:39.063822 6538 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:56:42.571825 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:42.571998 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:47.573251 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:47.573423 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:48.973105 6657 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:56:48.973274 6657 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:56:48.973834 6657 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:56:48.974862 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:56:48.976032 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:48.976168 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:48.976245 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:56:48.976428 6657 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:56:52.574577 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:52.574738 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:56.306991 6745 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:56:56.307157 6745 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:56:56.307714 6745 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:56:56.308624 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:56:56.309619 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:56.309756 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:56.309828 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:56:56.310030 6745 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:56:57.575912 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:56:57.576078 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:56:58.554826 6776 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:56:58.554980 6776 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:56:58.555438 6776 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:56:58.558269 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:56:58.558460 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:56:58.558569 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:56:58.558759 6776 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:02.577330 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:02.577484 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:07.578794 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:07.578953 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:57:09.362810 4029 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (186 consecutive failures): Network error: Failed to ping master at 127.3.77.62:34273: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:09.366829 4166 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (235 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:57:09.473038 4302 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (188 consecutive failures): Network error: Failed to ping master at 127.3.77.62:34273: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:57:12.580109 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:12.580271 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:16.649765 6984 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:57:16.649958 6984 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:57:16.650729 6984 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:57:16.651752 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 01:57:16.655194 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:16.655393 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:57:16.655489 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:57:16.655723 6984 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:17.581830 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:17.582044 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:22.583344 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:22.583523 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:27.584829 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:27.584985 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:29.262495 7141 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:57:29.262673 7141 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:57:29.263228 7141 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:57:29.264273 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:57:29.265815 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:29.265998 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:57:29.266076 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:57:29.266283 7141 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:32.379074 7178 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:57:32.379235 7178 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:57:32.379737 7178 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:57:32.383368 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:32.383567 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:57:32.383679 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:57:32.383926 7178 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:32.586103 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:32.586295 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:33.421994 7193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:57:33.422152 7193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:57:33.422750 7193 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:57:33.424872 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:33.425078 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:57:33.425161 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:57:33.425424 7193 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:37.555727 5719 maintenance_manager.cc:419] P 6ef804bcecd943d89c7af93e9375b1f3: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033350
I20250904 01:57:37.587512 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:37.587699 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:37.599226 5718 maintenance_manager.cc:643] P 6ef804bcecd943d89c7af93e9375b1f3: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.043s user 0.019s sys 0.004s Metrics: {"bytes_written":16020,"cfile_init":1,"dirs.queue_time_us":290,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":843,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":35,"thread_start_us":443,"threads_started":2,"wal-append.queue_time_us":318}
I20250904 01:57:37.600263 5719 maintenance_manager.cc:419] P 6ef804bcecd943d89c7af93e9375b1f3: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1928 bytes on disk
I20250904 01:57:37.600845 5718 maintenance_manager.cc:643] P 6ef804bcecd943d89c7af93e9375b1f3: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20250904 01:57:42.588924 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:42.589067 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:44.682641 7341 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:57:44.682821 7341 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:57:44.683419 7341 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:57:44.684494 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250904 01:57:44.686122 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:44.686317 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:57:44.686400 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:57:44.686604 7341 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:46.242493 7368 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:57:46.242671 7368 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:57:46.243155 7368 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:57:46.246491 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:46.246675 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:57:46.246778 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:57:46.247014 7368 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:47.590281 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:47.590432 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:52.591459 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:52.591605 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:53.618063 7489 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:57:53.618252 7489 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:57:53.618754 7489 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:57:53.619791 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 01:57:53.621002 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:57:53.621184 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:57:53.621296 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:57:53.621492 7489 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:57:57.592746 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:57:57.592905 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:02.594312 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:02.594465 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:05.244887 7673 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:58:05.245048 7673 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:58:05.245592 7673 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:58:05.246536 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:58:05.247829 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:58:05.247977 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:58:05.248057 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:58:05.248251 7673 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:58:07.595469 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:07.595620 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:58:09.523367 4029 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (306 consecutive failures): Network error: Failed to ping master at 127.3.77.62:34273: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:58:09.526997 4166 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (355 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:58:09.631978 4302 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (308 consecutive failures): Network error: Failed to ping master at 127.3.77.62:34273: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:58:10.089915 7754 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:58:10.090085 7754 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:58:10.090778 7754 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:58:10.092675 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:58:10.092828 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:58:10.092898 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:58:10.093102 7754 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:58:12.596918 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:12.597080 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:17.598292 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:17.598443 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:22.599552 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:22.599694 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:26.434906 7996 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:58:26.435083 7996 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:58:26.435832 7996 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:58:26.436841 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 01:58:26.439205 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:58:26.439378 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:58:26.439453 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:58:26.439699 7996 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:58:27.600662 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:27.600822 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:29.549944 8045 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:58:29.550140 8045 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:58:29.550791 8045 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:58:29.552691 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:58:29.552846 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:58:29.552947 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:58:29.553167 8045 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:58:32.601895 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:32.602041 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:37.603268 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:37.603442 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:42.604722 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:42.604882 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:44.649609 8276 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:58:44.649807 8276 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:58:44.650581 8276 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:58:44.651607 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 01:58:44.652981 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:58:44.653218 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 01:58:44.653311 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:58:44.653507 8276 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:58:47.606077 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:47.606298 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:52.607424 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:52.607616 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:57.608968 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:58:57.609141 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:00.204983 8504 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:00.205159 8504 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:00.205885 8504 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:00.206872 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:59:00.208451 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:00.208638 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:00.208729 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:00.208949 8504 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:59:02.610343 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:02.610508 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:03.471938 8548 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:03.472121 8548 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:03.472673 8548 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:03.476055 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:03.476248 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:03.476326 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:03.476569 8548 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:59:07.611768 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:07.611943 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:59:09.705412 4029 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (426 consecutive failures): Network error: Failed to ping master at 127.3.77.62:34273: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:09.706071 4166 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (475 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 01:59:09.798456 4301 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (476 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:12.613267 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:12.613435 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:13.235460 8690 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:13.235631 8690 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:13.236265 8690 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:13.237087 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 01:59:13.238131 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:13.238337 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:13.238437 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:13.238636 8690 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:59:17.614796 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:17.614964 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:18.762987 8769 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:18.763160 8769 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:18.763715 8769 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:18.764744 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:59:18.765858 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:18.766041 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:18.766112 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:18.766320 8769 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:59:22.616313 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:22.616490 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:27.617903 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:27.618055 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:32.619354 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:32.619524 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:34.152248 8986 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:34.152443 8986 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:34.153158 8986 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:34.154178 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:59:34.155318 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:34.155526 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:34.155611 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:34.155908 8986 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:59:37.620802 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:37.620962 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:41.815207 9096 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:41.815419 9096 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:41.816040 9096 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:41.818289 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:59:41.823343 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:41.823529 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:41.823611 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:41.823849 9096 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:59:42.622174 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:42.622383 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:47.623617 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:47.623769 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:52.625028 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:52.625211 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:56.829249 9303 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:56.829425 9303 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:56.830150 9303 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:56.831158 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 01:59:56.832183 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:56.832335 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:56.832394 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:56.832588 9303 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 01:59:57.626559 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 01:59:57.626752 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:59:57.942703 9328 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 01:59:57.942898 9328 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 01:59:57.943399 9328 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 01:59:57.945009 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 01:59:57.945166 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 01:59:57.945276 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 01:59:57.945497 9328 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 02:00:02.628472 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:02.628717 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:07.630046 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:07.630211 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:08.501953 9480 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 02:00:08.502128 9480 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 02:00:08.502756 9480 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 02:00:08.503715 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 02:00:08.504907 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 02:00:08.505074 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 02:00:08.505210 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 02:00:08.505463 9480 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250904 02:00:09.897899 4166 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (595 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 02:00:09.898408 4028 heartbeater.cc:646] Failed to heartbeat to 127.3.77.60:38761 (596 consecutive failures): Network error: Failed to ping master at 127.3.77.60:38761: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 02:00:09.955641 4302 heartbeater.cc:646] Failed to heartbeat to 127.3.77.62:34273 (548 consecutive failures): Network error: Failed to ping master at 127.3.77.62:34273: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
I20250904 02:00:12.631431 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:12.631599 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:17.632730 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:17.632906 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:18.886966 9625 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 02:00:18.887147 9625 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 02:00:18.887718 9625 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 02:00:18.888800 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 02:00:18.892787 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 02:00:18.893033 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 02:00:18.893146 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 02:00:18.893390 9625 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 02:00:22.634176 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:22.634374 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:26.092168 9729 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29b8a62cc8d04df781f991da2bd83ccc)
I20250904 02:00:26.092396 9729 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b8a62cc8d04df781f991da2bd83ccc" member_type: VOTER last_known_addr { host: "127.3.77.62" port: 34273 } } peers { permanent_uuid: "6ef804bcecd943d89c7af93e9375b1f3" member_type: VOTER last_known_addr { host: "127.3.77.61" port: 36307 } } peers { permanent_uuid: "7f298ba82a5e49028407388810de4b71" member_type: VOTER last_known_addr { host: "127.3.77.60" port: 38761 } }
I20250904 02:00:26.093120 9729 leader_election.cc:290] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273), 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761)
W20250904 02:00:26.094204 5714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 02:00:26.095698 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29b8a62cc8d04df781f991da2bd83ccc (127.3.77.62:34273): Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111)
W20250904 02:00:26.095916 5714 leader_election.cc:336] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f298ba82a5e49028407388810de4b71 (127.3.77.60:38761): Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 02:00:26.096029 5714 leader_election.cc:304] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef804bcecd943d89c7af93e9375b1f3; no voters: 29b8a62cc8d04df781f991da2bd83ccc, 7f298ba82a5e49028407388810de4b71
I20250904 02:00:26.096279 9729 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6ef804bcecd943d89c7af93e9375b1f3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 02:00:27.635622 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:27.635775 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:32.636950 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:32.637111 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:37.638401 5697 master_runner.cc:186] Error getting master registration for 127.3.77.62:34273: Network error: Client connection negotiation failed: client connection to 127.3.77.62:34273: connect: Connection refused (error 111),
I20250904 02:00:37.638566 5697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 02:00:38.073057 5719 maintenance_manager.cc:419] P 6ef804bcecd943d89c7af93e9375b1f3: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1928 bytes on disk
W20250904 02:00:38.075851 4309 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 02:00:38.076176 4309 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 02:00:38.076284 4309 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
W20250904 02:00:38.076391 4309 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
I20250904 02:00:38.078616 5718 maintenance_manager.cc:643] P 6ef804bcecd943d89c7af93e9375b1f3: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":272,"thread_start_us":362,"threads_started":2,"wal-append.queue_time_us":275}
F20250904 02:00:38.087191 4330 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1756951238 (unix time) try "date -d @1756951238" if you are using GNU date ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000d34) received by PID 3380 (TID 0x7fe845a5a700) from PID 3380; stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095534 4334 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fe85f40c980 (unknown) at ??:0
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095534 4334 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095639 4323 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_338af7b0a5ed48918a49777a432224f9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095534 4334 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095639 4323 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_338af7b0a5ed48918a49777a432224f9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096047 4337 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095534 4334 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095639 4323 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_338af7b0a5ed48918a49777a432224f9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096047 4337 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096071 4317 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bce07055a7064408b762f1534d59a90e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095534 4334 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095639 4323 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_338af7b0a5ed48918a49777a432224f9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096047 4337 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096071 4317 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bce07055a7064408b762f1534d59a90e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096309 4329 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095534 4334 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095639 4323 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_338af7b0a5ed48918a49777a432224f9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096047 4337 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096071 4317 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bce07055a7064408b762f1534d59a90e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096309 4329 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096402 4342 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 02:00:38.087206 4332 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088330 4328 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.088390 4327 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089782 4316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c245a71ea8e74145959886fcfac66f07 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.089881 4322 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7df3fb23f02e4d4c8bcf74a0350c7aac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090243 4325 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bc5b38d3266841d6b4a43d4983cff6b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090245 4318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d723b3eb51ff4772a7331c9338cb1918 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090376 4320 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_39d410e70d9349abad85c4dccbad54bf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090387 4333 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.090606 4321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d03c97312adf4523be9539809e8fdb5d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091641 4336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.091684 4340 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.092829 4326 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.093026 4319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_968a17a9054a48dda6494188adb73474 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.094192 4341 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out @ 0x7fe8583d2fb7 gsignal
: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095460 4324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_782e3736e3434cafa6e2825f14c5900b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095534 4334 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.095639 4323 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_338af7b0a5ed48918a49777a432224f9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096047 4337 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096071 4317 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bce07055a7064408b762f1534d59a90e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096309 4329 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096402 4342 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)F20250904 02:00:38.096495 4331 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.77.62:34273,127.3.77.61:36307,127.3.77.60:38761, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.77.60:38761: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fe8583d4921 abort at ??:0
@ 0x7fe85b10adcd google::LogMessage::Fail() at ??:0
@ 0x7fe85b10eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe85b10a7cc google::LogMessage::Flush() at ??:0
@ 0x7fe85b10bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40c481 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:328
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fe858a5a6df (unknown) at ??:0
@ 0x7fe85f4016db start_thread at ??:0
@ 0x7fe8584b571f clone at ??:0