Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-01-29T14:55:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-29T14:55:29Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskf9Nfvs/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskf9Nfvs/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/security.properties
2026-01-29 14:55:35: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskf9Nfvs/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskf9Nfvs/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260129 14:55:43.171669 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--webserver_interface=127.7.171.254
--webserver_port=0
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:55:43.473184 8096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:43.473466 8096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:43.473537 8096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:43.481252 8096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:55:43.481329 8096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:43.481393 8096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:55:43.481447 8096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:55:43.490211 8096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.171.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:43.491662 8096 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:43.493309 8096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:55:43.501446 8102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:43.501422 8101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:43.501840 8104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:43.502353 8096 server_base.cc:1060] running on GCE node
I20260129 14:55:43.502992 8096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:43.504269 8096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:43.505460 8096 hybrid_clock.cc:648] HybridClock initialized: now 1769698543505388 us; error 67 us; skew 500 ppm
I20260129 14:55:43.505779 8096 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:43.507596 8096 webserver.cc:492] Webserver started at http://127.7.171.254:33955/ using document root <none> and password file <none>
I20260129 14:55:43.508280 8096 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:43.508379 8096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:43.508683 8096 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 14:55:43.510763 8096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/instance:
uuid: "5cde2b69b78d4e8491572266f7019c6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:43.511653 8096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal/instance:
uuid: "5cde2b69b78d4e8491572266f7019c6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:43.516377 8096 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260129 14:55:43.519359 8110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:43.520484 8096 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 14:55:43.520624 8096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
uuid: "5cde2b69b78d4e8491572266f7019c6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:43.520820 8096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:43.542469 8096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:43.543154 8096 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:43.543377 8096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:43.561058 8096 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.254:38695
I20260129 14:55:43.561092 8161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.254:38695 every 8 connection(s)
I20260129 14:55:43.562352 8096 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
I20260129 14:55:43.565930 8162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:43.566998 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 8096
I20260129 14:55:43.567256 7855 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal/instance
I20260129 14:55:43.569676 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.253:34637
--webserver_interface=127.7.171.253
--webserver_port=0
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260129 14:55:43.571377 8162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260129 14:55:43.585155 8162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
W20260129 14:55:43.586112 8111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:43.587399 8162 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:43.636826 8162 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } attempt: 1
W20260129 14:55:43.638496 8162 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:43.706789 8162 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } attempt: 2
W20260129 14:55:43.708618 8162 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:43.799827 8162 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } attempt: 3
W20260129 14:55:43.801563 8162 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:43.871642 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:43.871964 8163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:43.872032 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:43.880402 8163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:55:43.880496 8163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:43.880549 8163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:55:43.880597 8163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:55:43.890300 8163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.253:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.171.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:43.891855 8163 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:43.893518 8163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:55:43.901368 8171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:43.901329 8172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:43.902578 8174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:43.902999 8163 server_base.cc:1060] running on GCE node
I20260129 14:55:43.903666 8163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:43.904984 8163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:43.906191 8163 hybrid_clock.cc:648] HybridClock initialized: now 1769698543906146 us; error 39 us; skew 500 ppm
I20260129 14:55:43.906522 8163 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:43.908480 8163 webserver.cc:492] Webserver started at http://127.7.171.253:41757/ using document root <none> and password file <none>
I20260129 14:55:43.909190 8163 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:43.909292 8163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:43.909617 8163 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 14:55:43.913022 8163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/instance:
uuid: "c63d8c4ffa4047c9a8423771f4723d6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:43.913765 8163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal/instance:
uuid: "c63d8c4ffa4047c9a8423771f4723d6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:43.918763 8163 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260129 14:55:43.921794 8180 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:43.923004 8163 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 14:55:43.923161 8163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
uuid: "c63d8c4ffa4047c9a8423771f4723d6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:43.923342 8163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:43.939199 8163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:43.939965 8163 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:43.940227 8163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:43.944764 8162 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } attempt: 4
W20260129 14:55:43.946542 8162 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:43.959069 8163 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.253:34637
I20260129 14:55:43.959086 8231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.253:34637 every 8 connection(s)
I20260129 14:55:43.960495 8163 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
I20260129 14:55:43.964286 8232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:43.964802 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 8163
I20260129 14:55:43.965018 7855 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal/instance
I20260129 14:55:43.966396 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--webserver_interface=127.7.171.252
--webserver_port=0
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260129 14:55:43.972067 8232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260129 14:55:43.983853 8232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
I20260129 14:55:43.991739 8232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } has no permanent_uuid. Determining permanent_uuid...
W20260129 14:55:43.992520 8182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
W20260129 14:55:43.993830 8232 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:44.043165 8232 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } attempt: 1
W20260129 14:55:44.045012 8232 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:44.113286 8232 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } attempt: 2
W20260129 14:55:44.115120 8232 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:44.206395 8232 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } attempt: 3
W20260129 14:55:44.208128 8232 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:44.245966 8162 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } attempt: 5
I20260129 14:55:44.252267 8162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } has no permanent_uuid. Determining permanent_uuid...
W20260129 14:55:44.253824 8162 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
W20260129 14:55:44.278301 8233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:44.278556 8233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:44.278615 8233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:44.286155 8233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:55:44.286239 8233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:44.286304 8233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:55:44.286338 8233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:55:44.296563 8233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.171.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:44.298373 8233 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:44.300400 8233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 14:55:44.305003 8162 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } attempt: 1
W20260129 14:55:44.307118 8162 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
W20260129 14:55:44.307775 8241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:44.307806 8242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:44.313277 8244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:44.313611 8233 server_base.cc:1060] running on GCE node
I20260129 14:55:44.314237 8233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:44.315464 8233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:44.316664 8233 hybrid_clock.cc:648] HybridClock initialized: now 1769698544316620 us; error 29 us; skew 500 ppm
I20260129 14:55:44.316987 8233 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:44.318760 8233 webserver.cc:492] Webserver started at http://127.7.171.252:37043/ using document root <none> and password file <none>
I20260129 14:55:44.319365 8233 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:44.319458 8233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:44.319787 8233 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 14:55:44.321938 8233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/instance:
uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
format_stamp: "Formatted at 2026-01-29 14:55:44 on dist-test-slave-n0m6"
I20260129 14:55:44.322599 8233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal/instance:
uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
format_stamp: "Formatted at 2026-01-29 14:55:44 on dist-test-slave-n0m6"
I20260129 14:55:44.327172 8233 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260129 14:55:44.329941 8250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:44.331135 8233 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260129 14:55:44.331298 8233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
format_stamp: "Formatted at 2026-01-29 14:55:44 on dist-test-slave-n0m6"
I20260129 14:55:44.331516 8233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:44.351358 8232 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } attempt: 4
W20260129 14:55:44.353379 8232 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:44.354877 8233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:44.355643 8233 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:44.355854 8233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:44.373876 8233 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.252:43411
I20260129 14:55:44.373893 8301 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.252:43411 every 8 connection(s)
I20260129 14:55:44.375178 8233 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
I20260129 14:55:44.375342 8162 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } attempt: 2
I20260129 14:55:44.379122 8302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:44.383265 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 8233
I20260129 14:55:44.383438 7855 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal/instance
I20260129 14:55:44.385241 8302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260129 14:55:44.392284 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap starting.
I20260129 14:55:44.396994 8162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:44.398687 8162 log.cc:826] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:44.403350 8302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
I20260129 14:55:44.407855 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: No bootstrap required, opened a new log
I20260129 14:55:44.408712 8302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } has no permanent_uuid. Determining permanent_uuid...
I20260129 14:55:44.414459 8162 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.414846 8162 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:44.415035 8162 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Initialized, Role: FOLLOWER
I20260129 14:55:44.415776 8162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.417213 8310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.417572 8310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:44.418299 8162 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:55:44.420553 8302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap starting.
I20260129 14:55:44.424022 8302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:44.425387 8302 log.cc:826] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:44.428197 8302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: No bootstrap required, opened a new log
I20260129 14:55:44.433107 8302 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.433333 8302 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:44.433442 8302 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db7ebac21af437fbdf8e1bd7e00fca2, State: Initialized, Role: FOLLOWER
I20260129 14:55:44.434006 8302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.434792 8322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.435124 8322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:44.435680 8302 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: configured and running, proceeding with master startup.
W20260129 14:55:44.449481 8326 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260129 14:55:44.449671 8326 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260129 14:55:44.441327 8096 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: OK, instance_id { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" instance_seqno: 1769698543954143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260129 14:55:44.450536 8096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:55:44.473603 8310 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:44.473897 8310 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.475749 8310 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:55:44.477694 8277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" is_pre_election: true
I20260129 14:55:44.478231 8277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 0.
I20260129 14:55:44.478874 8112 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters:
I20260129 14:55:44.479262 8310 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 14:55:44.479416 8310 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:55:44.479519 8310 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:44.482733 8310 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.486739 8277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
I20260129 14:55:44.486951 8277 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:44.481086 8233 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: OK, instance_id { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" instance_seqno: 1769698543954143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260129 14:55:44.487437 8233 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:55:44.489933 8277 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 1.
I20260129 14:55:44.490343 8310 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 1 election: Requested vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:55:44.490396 8112 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters:
I20260129 14:55:44.491078 8310 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260129 14:55:44.491856 8310 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 1 LEADER]: Becoming Leader. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Running, Role: LEADER
I20260129 14:55:44.493459 8310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
W20260129 14:55:44.494299 8343 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260129 14:55:44.494452 8343 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260129 14:55:44.509176 8345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde2b69b78d4e8491572266f7019c6e. Latest consensus state: current_term: 1 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.509574 8345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:44.515350 8347 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:55:44.517638 8347 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:55:44.531497 8277 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Refusing update from remote peer 5cde2b69b78d4e8491572266f7019c6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 14:55:44.532523 8345 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260129 14:55:44.545773 8116 tablet.cc:2378] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260129 14:55:44.554951 8322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde2b69b78d4e8491572266f7019c6e. Latest consensus state: current_term: 1 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.555189 8322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:44.559134 8345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.559443 8345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:44.561108 8349 mvcc.cc:204] Tried to move back new op lower bound from 7248685238392066048 to 7248685238264913920. Current Snapshot: MvccSnapshot[applied={T|T < 7248685238392066048}]
I20260129 14:55:44.568543 8322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.568745 8322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:44.591029 8347 catalog_manager.cc:1357] Generated new cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:44.591147 8347 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:55:44.652582 8232 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } attempt: 5
I20260129 14:55:44.663101 8347 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 14:55:44.664973 8347 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:55:44.669377 8232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Bootstrap starting.
I20260129 14:55:44.677208 8232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:44.679626 8232 log.cc:826] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:44.696717 8347 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Generated new TSK 0
I20260129 14:55:44.697947 8347 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:55:44.699216 8347 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:55:44.701237 8232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: No bootstrap required, opened a new log
I20260129 14:55:44.702594 8347 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:55:44.709793 8232 raft_consensus.cc:359] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.710318 8232 raft_consensus.cc:385] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:44.710456 8232 raft_consensus.cc:740] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63d8c4ffa4047c9a8423771f4723d6e, State: Initialized, Role: FOLLOWER
I20260129 14:55:44.711179 8232 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:44.711570 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.171.193:0
--local_ip_for_outbound_sockets=127.7.171.193
--webserver_interface=127.7.171.193
--webserver_port=0
--tserver_master_addrs=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260129 14:55:44.712908 8354 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.713580 8354 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:44.714195 8232 sys_catalog.cc:565] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:55:44.714553 8205 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:44.718883 8205 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Refusing update from remote peer 5cde2b69b78d4e8491572266f7019c6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 14:55:44.719995 8310 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:55:44.720845 8207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" is_pre_election: true
I20260129 14:55:44.721400 8207 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 1.
I20260129 14:55:44.721760 8206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63d8c4ffa4047c9a8423771f4723d6e"
I20260129 14:55:44.727769 8206 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 1.
I20260129 14:55:44.761781 8354 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde2b69b78d4e8491572266f7019c6e. Latest consensus state: current_term: 1 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.762084 8354 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:44.762404 8359 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:44.762738 8359 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:44.824342 8376 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:44.824497 8376 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: loading cluster ID for follower catalog manager: success
I20260129 14:55:44.828142 8376 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: acquiring CA information for follower catalog manager: success
I20260129 14:55:44.830283 8376 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260129 14:55:45.196105 8356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:45.196409 8356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:45.196570 8356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:45.204815 8356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:45.205035 8356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.171.193
I20260129 14:55:45.214421 8356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.171.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.171.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.171.193
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:45.215929 8356 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:45.217614 8356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 14:55:45.222420 8310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:45.222683 8310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: LEADER
W20260129 14:55:45.237241 8386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:45.237807 8385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:45.238546 8388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:45.239068 8356 server_base.cc:1060] running on GCE node
I20260129 14:55:45.239617 8356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:45.240653 8356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:45.241863 8356 hybrid_clock.cc:648] HybridClock initialized: now 1769698545241822 us; error 37 us; skew 500 ppm
I20260129 14:55:45.242161 8356 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:45.244252 8356 webserver.cc:492] Webserver started at http://127.7.171.193:46071/ using document root <none> and password file <none>
I20260129 14:55:45.244932 8356 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:45.245018 8356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:45.245411 8356 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 14:55:45.247562 8356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data/instance:
uuid: "824b258cd009461bb4f25beea2af6eb9"
format_stamp: "Formatted at 2026-01-29 14:55:45 on dist-test-slave-n0m6"
I20260129 14:55:45.248347 8356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal/instance:
uuid: "824b258cd009461bb4f25beea2af6eb9"
format_stamp: "Formatted at 2026-01-29 14:55:45 on dist-test-slave-n0m6"
I20260129 14:55:45.253089 8356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260129 14:55:45.256232 8394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:45.257418 8356 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260129 14:55:45.257550 8356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal
uuid: "824b258cd009461bb4f25beea2af6eb9"
format_stamp: "Formatted at 2026-01-29 14:55:45 on dist-test-slave-n0m6"
I20260129 14:55:45.257741 8356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:45.275372 8356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:45.276177 8356 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:45.276481 8356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:45.277429 8356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 14:55:45.279102 8356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 14:55:45.279188 8356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:45.279294 8356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 14:55:45.279368 8356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:45.319097 8356 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.193:44151
I20260129 14:55:45.319128 8506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.193:44151 every 8 connection(s)
I20260129 14:55:45.320652 8356 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/data/info.pb
I20260129 14:55:45.324644 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 8356
I20260129 14:55:45.324888 7855 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal/instance
I20260129 14:55:45.336802 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.171.194:0
--local_ip_for_outbound_sockets=127.7.171.194
--webserver_interface=127.7.171.194
--webserver_port=0
--tserver_master_addrs=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260129 14:55:45.386443 8507 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:55:45.386818 8507 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:45.387759 8507 heartbeater.cc:507] Master 127.7.171.252:43411 requested a full tablet report, sending...
I20260129 14:55:45.390344 8267 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:45.393023 8508 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:45.393127 8508 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:45.393548 8508 heartbeater.cc:507] Master 127.7.171.254:38695 requested a full tablet report, sending...
I20260129 14:55:45.395126 8127 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:45.397265 8127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.171.193:37801
I20260129 14:55:45.404533 8512 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:55:45.404682 8512 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:45.404996 8512 heartbeater.cc:507] Master 127.7.171.253:34637 requested a full tablet report, sending...
I20260129 14:55:45.406339 8197 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:45.496805 8343 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:45.496933 8343 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: loading cluster ID for follower catalog manager: success
I20260129 14:55:45.500420 8343 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: acquiring CA information for follower catalog manager: success
I20260129 14:55:45.502526 8343 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260129 14:55:45.695274 8511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:45.695554 8511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:45.695688 8511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:45.704502 8511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:45.704746 8511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.171.194
I20260129 14:55:45.714505 8511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.171.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.171.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.171.194
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:45.716094 8511 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:45.717831 8511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:55:45.727099 8523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:45.727066 8522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:45.727401 8525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:45.728533 8511 server_base.cc:1060] running on GCE node
I20260129 14:55:45.729063 8511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:45.729893 8511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:45.731069 8511 hybrid_clock.cc:648] HybridClock initialized: now 1769698545731023 us; error 46 us; skew 500 ppm
I20260129 14:55:45.731326 8511 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:45.733198 8511 webserver.cc:492] Webserver started at http://127.7.171.194:43985/ using document root <none> and password file <none>
I20260129 14:55:45.733880 8511 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:45.733966 8511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:45.734301 8511 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 14:55:45.737098 8511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data/instance:
uuid: "0721a8132fc84354ae4b9e42fe387c67"
format_stamp: "Formatted at 2026-01-29 14:55:45 on dist-test-slave-n0m6"
I20260129 14:55:45.737802 8511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal/instance:
uuid: "0721a8132fc84354ae4b9e42fe387c67"
format_stamp: "Formatted at 2026-01-29 14:55:45 on dist-test-slave-n0m6"
I20260129 14:55:45.742313 8511 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 14:55:45.745517 8531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:45.746728 8511 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260129 14:55:45.746891 8511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal
uuid: "0721a8132fc84354ae4b9e42fe387c67"
format_stamp: "Formatted at 2026-01-29 14:55:45 on dist-test-slave-n0m6"
I20260129 14:55:45.747068 8511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:45.765873 8511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:45.766629 8511 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:45.766860 8511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:45.767693 8511 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 14:55:45.769212 8511 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 14:55:45.769305 8511 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:45.769412 8511 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 14:55:45.769480 8511 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:45.803625 8511 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.194:44263
I20260129 14:55:45.803663 8643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.194:44263 every 8 connection(s)
I20260129 14:55:45.804868 8511 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/data/info.pb
I20260129 14:55:45.808703 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 8511
I20260129 14:55:45.808928 7855 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal/instance
I20260129 14:55:45.813766 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.171.195:0
--local_ip_for_outbound_sockets=127.7.171.195
--webserver_interface=127.7.171.195
--webserver_port=0
--tserver_master_addrs=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260129 14:55:45.821717 8645 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:45.822126 8645 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:45.822264 8646 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:55:45.822440 8646 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:45.823279 8645 heartbeater.cc:507] Master 127.7.171.254:38695 requested a full tablet report, sending...
I20260129 14:55:45.823304 8646 heartbeater.cc:507] Master 127.7.171.253:34637 requested a full tablet report, sending...
I20260129 14:55:45.824710 8127 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:45.825349 8197 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:45.825858 8644 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:55:45.825855 8127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.171.194:51121
I20260129 14:55:45.825965 8644 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:45.826435 8644 heartbeater.cc:507] Master 127.7.171.252:43411 requested a full tablet report, sending...
I20260129 14:55:45.827194 8267 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:45.899722 8508 heartbeater.cc:499] Master 127.7.171.254:38695 was elected leader, sending a full tablet report...
I20260129 14:55:46.125510 8652 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:55:46.125707 8652 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:46.126947 8652 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
I20260129 14:55:46.127566 8207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" is_pre_election: true
I20260129 14:55:46.128191 8137 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e" is_pre_election: true
I20260129 14:55:46.128880 8252 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:46.129194 8652 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260129 14:55:46.141727 8651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:46.142032 8651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:46.142154 8651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:46.149972 8651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:46.150174 8651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.171.195
I20260129 14:55:46.160292 8651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.171.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.171.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.171.195
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:46.161827 8651 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:46.163513 8651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 14:55:46.173442 8651 server_base.cc:1060] running on GCE node
W20260129 14:55:46.173581 8660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:46.173825 8657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:46.173887 8658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:46.174486 8651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:46.175235 8651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:46.176443 8651 hybrid_clock.cc:648] HybridClock initialized: now 1769698546176380 us; error 39 us; skew 500 ppm
I20260129 14:55:46.176712 8651 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:46.178644 8651 webserver.cc:492] Webserver started at http://127.7.171.195:39287/ using document root <none> and password file <none>
I20260129 14:55:46.179296 8651 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:46.179395 8651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:46.179719 8651 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 14:55:46.181919 8651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data/instance:
uuid: "e6f177394443439dbabb0738a08b8fc3"
format_stamp: "Formatted at 2026-01-29 14:55:46 on dist-test-slave-n0m6"
I20260129 14:55:46.182657 8651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal/instance:
uuid: "e6f177394443439dbabb0738a08b8fc3"
format_stamp: "Formatted at 2026-01-29 14:55:46 on dist-test-slave-n0m6"
I20260129 14:55:46.187634 8651 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260129 14:55:46.191257 8666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:46.192477 8651 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260129 14:55:46.192629 8651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal
uuid: "e6f177394443439dbabb0738a08b8fc3"
format_stamp: "Formatted at 2026-01-29 14:55:46 on dist-test-slave-n0m6"
I20260129 14:55:46.192800 8651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:46.204945 8651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:46.205739 8651 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:46.206018 8651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:46.206807 8651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 14:55:46.208287 8651 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 14:55:46.208385 8651 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:46.208503 8651 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 14:55:46.208582 8651 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:46.243582 8651 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.195:44395
I20260129 14:55:46.243607 8778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.195:44395 every 8 connection(s)
I20260129 14:55:46.244969 8651 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/data/info.pb
I20260129 14:55:46.245234 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 8651
I20260129 14:55:46.245409 7855 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal/instance
I20260129 14:55:46.270931 8780 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:46.271265 8780 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:46.271873 8781 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:55:46.272002 8781 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:46.272257 8779 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:55:46.272343 8780 heartbeater.cc:507] Master 127.7.171.254:38695 requested a full tablet report, sending...
I20260129 14:55:46.272385 8779 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:46.272528 8781 heartbeater.cc:507] Master 127.7.171.253:34637 requested a full tablet report, sending...
I20260129 14:55:46.272915 8779 heartbeater.cc:507] Master 127.7.171.252:43411 requested a full tablet report, sending...
I20260129 14:55:46.273792 8127 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:46.273921 8267 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:46.274139 8196 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:46.274833 8127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.171.195:42101
I20260129 14:55:46.288666 7855 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260129 14:55:46.325129 8127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.325860 8126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.326223 8125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.326428 8124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.327802 8123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_60640f05a979404380083e385f1e6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.328181 8645 heartbeater.cc:499] Master 127.7.171.254:38695 was elected leader, sending a full tablet report...
I20260129 14:55:46.328747 8121 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.328775 8120 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.329450 8122 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_17c52877671f4e30bcf4f682122e181e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.329488 8119 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_c2b80faabf21435d991d592f638440aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.325927 8118 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45484:
name: "default.Table_6ecb7f939f3648ebab36e0f73528521a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260129 14:55:46.341984 8127 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_29883fd0b006400b953426f944f08202 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.343899 8126 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_95ece788510d4811b741bfafc484dd89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 14:55:46.364975 7855 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu pid 8096
I20260129 14:55:46.366235 8157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260129 14:55:46.517460 7855 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu with pid 8096
W20260129 14:55:46.541635 8645 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.7.171.254:38695 (error 108)
W20260129 14:55:46.542094 8787 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.171.254:38695)
W20260129 14:55:46.542371 8508 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.7.171.254:38695 (error 108)
W20260129 14:55:46.545388 8792 negotiation.cc:337] Failed RPC negotiation. Trace:
0129 14:55:46.543341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.171.254:38695 (local address 127.0.0.1:45496)
0129 14:55:46.543698 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0129 14:55:46.543914 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":89}
I20260129 14:55:46.578980 8826 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:55:46.579267 8826 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:46.580842 8826 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:55:46.594825 8277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" is_pre_election: true
W20260129 14:55:46.601173 8182 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:55:46.601348 8182 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c63d8c4ffa4047c9a8423771f4723d6e; no voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2
I20260129 14:55:46.601742 8826 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 14:55:46.663756 8829 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:55:46.664110 8829 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:46.667037 8829 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
I20260129 14:55:46.667104 8207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" is_pre_election: true
I20260129 14:55:46.667335 8207 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db7ebac21af437fbdf8e1bd7e00fca2 in term 1.
I20260129 14:55:46.668013 8251 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e; no voters:
I20260129 14:55:46.668411 8829 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 14:55:46.668560 8829 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:55:46.668702 8829 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 1 FOLLOWER]: Advancing to term 2
W20260129 14:55:46.669787 8252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:55:46.671825 8829 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:46.673106 8207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c63d8c4ffa4047c9a8423771f4723d6e"
I20260129 14:55:46.673321 8207 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 1 FOLLOWER]: Advancing to term 2
I20260129 14:55:46.676028 8207 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db7ebac21af437fbdf8e1bd7e00fca2 in term 2.
I20260129 14:55:46.676527 8251 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e; no voters:
I20260129 14:55:46.680286 8829 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 election: Requested vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
I20260129 14:55:46.680676 8829 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 14:55:46.680958 8829 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 LEADER]: Becoming Leader. State: Replica: 8db7ebac21af437fbdf8e1bd7e00fca2, State: Running, Role: LEADER
I20260129 14:55:46.681500 8829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
W20260129 14:55:46.686339 8252 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:55:46.687669 8252 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:55:46.688139 8829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db7ebac21af437fbdf8e1bd7e00fca2. Latest consensus state: current_term: 2 leader_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:46.688359 8829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:46.689020 8834 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:55:46.691498 8834 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:55:46.692569 8834 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:46.692652 8834 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:55:46.694108 8834 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:55:46.695106 8834 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Loaded TSK: 0
I20260129 14:55:46.696338 8834 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:55:46.697413 8834 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:55:46.698423 8834 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:55:46.707504 8267 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_6ecb7f939f3648ebab36e0f73528521a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.707561 8266 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_c2b80faabf21435d991d592f638440aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.708245 8261 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.708375 8264 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.708878 8263 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.708951 8259 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_60640f05a979404380083e385f1e6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.709020 8262 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.709632 8258 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_17c52877671f4e30bcf4f682122e181e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.708427 8265 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:46.710232 8260 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260129 14:55:46.718007 8266 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c2b80faabf21435d991d592f638440aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.720333 8267 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6ecb7f939f3648ebab36e0f73528521a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 14:55:46.776607 8780 heartbeater.cc:499] Master 127.7.171.254:38695 was elected leader, sending a full tablet report...
W20260129 14:55:46.778010 8780 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:55:46.831347 8266 catalog_manager.cc:2174] created HMS catalog entry for table [id=908853f113ee48df891ae3a9db09c1ed]
I20260129 14:55:46.835441 8207 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Refusing update from remote peer 8db7ebac21af437fbdf8e1bd7e00fca2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260129 14:55:46.836135 8832 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260129 14:55:46.836701 8252 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 -> Peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Couldn't send request to peer 5cde2b69b78d4e8491572266f7019c6e. Status: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 14:55:46.860572 8826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db7ebac21af437fbdf8e1bd7e00fca2. Latest consensus state: current_term: 2 leader_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:46.860800 8826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:46.862478 8829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:46.862701 8829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:46.862987 8837 mvcc.cc:204] Tried to move back new op lower bound from 7248685247831433216 to 7248685247212048384. Current Snapshot: MvccSnapshot[applied={T|T < 7248685247831433216}]
I20260129 14:55:46.867000 8826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:46.867204 8826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:46.869837 8267 catalog_manager.cc:2174] created HMS catalog entry for table [id=b78f947e473549d9a57b0b87c8cc14fc]
I20260129 14:55:46.871522 8266 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59680:
table { table_name: "default.Table_c2b80faabf21435d991d592f638440aa" } new_table_name: "default.Table_2ec3463f89a9421cac3c15af609890f3" modify_external_catalogs: true
W20260129 14:55:46.872690 8261 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_acb782ad01664cc1a5a6ba4cb226d899 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.872722 8260 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_663408aa31d3405ab53f9afa189ceeaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.873512 8264 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a0a36d2e9fd749c9958b59c392a0e64c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.874421 8263 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_95ece788510d4811b741bfafc484dd89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.875398 8259 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_60640f05a979404380083e385f1e6ae0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.876284 8258 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_17c52877671f4e30bcf4f682122e181e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.877266 8265 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260129 14:55:46.878749 8262 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_29883fd0b006400b953426f944f08202 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
F20260129 14:55:46.890609 8341 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_c2b80faabf21435d991d592f638440aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2b80faabf21435d991d592f638440aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2b80faabf21435d991d592f638440aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_c2b80faabf21435d991d592f638440aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2b80faabf21435d991d592f638440aa\",\"kudu.table_id\":\"908853f113ee48df891ae3a9db09c1ed\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_95ece788510d4811b741bfafc484dd89, message={"server":"","servicePrincipal":"","db":"default","table":"table_95ece788510d4811b741bfafc484dd89","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_95ece788510d4811b741bfafc484dd89\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_95ece788510d4811b741bfafc484dd89\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_95ece788510d4811b741bfafc484dd89\",\"kudu.table_id\":\"66667881263c4bb0a181c0c7576354f7\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fb8a8fb4dcd google::LogMessage::Fail() at ??:0
@ 0x7fb8a8fb8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb8a8fb47cc google::LogMessage::Flush() at ??:0
@ 0x7fb8a8fb4a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb8bf9ab7f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260129 14:55:46.924870 8267 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59680:
table { table_name: "default.Table_6ecb7f939f3648ebab36e0f73528521a" } new_table_name: "default.Table_8c3d7c81c8d148a685a90b32620410a9" modify_external_catalogs: true
@ 0x7fb8bf9aa0a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260129 14:55:46.931085 8714 tablet_service.cc:1505] Processing CreateTablet for tablet 7b5bc5c45b2a4314b92c797aac0ca371 (DEFAULT_TABLE table=default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:46.932689 8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5bc5c45b2a4314b92c797aac0ca371. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb8bf9b1d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260129 14:55:46.940102 8442 tablet_service.cc:1505] Processing CreateTablet for tablet 7b5bc5c45b2a4314b92c797aac0ca371 (DEFAULT_TABLE table=default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:46.941546 8442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5bc5c45b2a4314b92c797aac0ca371. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb8bf9b1b99 std::_Function_handler<>::_M_invoke() at ??:0
I20260129 14:55:46.944970 8441 tablet_service.cc:1505] Processing CreateTablet for tablet d7c50ee4a0464ec4a03a6a30e4cb74df (DEFAULT_TABLE table=default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0xa8970d std::function<>::operator()() at ??:?
I20260129 14:55:46.945628 8441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c50ee4a0464ec4a03a6a30e4cb74df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:46.952363 8713 tablet_service.cc:1505] Processing CreateTablet for tablet d7c50ee4a0464ec4a03a6a30e4cb74df (DEFAULT_TABLE table=default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:46.952847 8713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c50ee4a0464ec4a03a6a30e4cb74df. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb8abd29f99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb8af3366db start_thread at ??:0
@ 0x7fb8a7e6571f clone at ??:0
I20260129 14:55:46.967774 8853 tablet_bootstrap.cc:492] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Bootstrap starting.
I20260129 14:55:46.968380 8579 tablet_service.cc:1505] Processing CreateTablet for tablet 7b5bc5c45b2a4314b92c797aac0ca371 (DEFAULT_TABLE table=default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:46.969200 8851 tablet_bootstrap.cc:492] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Bootstrap starting.
I20260129 14:55:46.969741 8579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5bc5c45b2a4314b92c797aac0ca371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:46.971078 8853 tablet_bootstrap.cc:654] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:46.972267 8851 tablet_bootstrap.cc:654] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:46.968004 8578 tablet_service.cc:1505] Processing CreateTablet for tablet d7c50ee4a0464ec4a03a6a30e4cb74df (DEFAULT_TABLE table=default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:46.972729 8578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c50ee4a0464ec4a03a6a30e4cb74df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:46.973388 8851 log.cc:826] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:46.984949 8853 log.cc:826] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:47.006922 8853 tablet_bootstrap.cc:492] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: No bootstrap required, opened a new log
I20260129 14:55:47.007208 8853 ts_tablet_manager.cc:1403] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.007s
I20260129 14:55:47.011696 8859 tablet_bootstrap.cc:492] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Bootstrap starting.
I20260129 14:55:47.023314 8859 tablet_bootstrap.cc:654] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:47.024746 8859 log.cc:826] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:47.027602 8851 tablet_bootstrap.cc:492] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: No bootstrap required, opened a new log
I20260129 14:55:47.027866 8851 ts_tablet_manager.cc:1403] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Time spent bootstrapping tablet: real 0.059s user 0.032s sys 0.018s
I20260129 14:55:47.029690 8853 raft_consensus.cc:359] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.030067 8853 raft_consensus.cc:385] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:47.030157 8853 raft_consensus.cc:740] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824b258cd009461bb4f25beea2af6eb9, State: Initialized, Role: FOLLOWER
I20260129 14:55:47.030843 8853 consensus_queue.cc:260] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.031793 8507 heartbeater.cc:499] Master 127.7.171.252:43411 was elected leader, sending a full tablet report...
I20260129 14:55:47.037405 8853 ts_tablet_manager.cc:1434] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Time spent starting tablet: real 0.030s user 0.009s sys 0.000s
I20260129 14:55:47.037885 8853 tablet_bootstrap.cc:492] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Bootstrap starting.
I20260129 14:55:47.039669 8853 tablet_bootstrap.cc:654] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:47.041563 8859 tablet_bootstrap.cc:492] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: No bootstrap required, opened a new log
I20260129 14:55:47.041808 8853 tablet_bootstrap.cc:492] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: No bootstrap required, opened a new log
I20260129 14:55:47.041826 8859 ts_tablet_manager.cc:1403] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.001s
I20260129 14:55:47.041924 8853 ts_tablet_manager.cc:1403] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260129 14:55:47.042414 8851 raft_consensus.cc:359] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.042690 8853 raft_consensus.cc:359] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.042810 8851 raft_consensus.cc:385] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:47.042878 8853 raft_consensus.cc:385] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:47.042896 8851 raft_consensus.cc:740] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f177394443439dbabb0738a08b8fc3, State: Initialized, Role: FOLLOWER
I20260129 14:55:47.042956 8853 raft_consensus.cc:740] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824b258cd009461bb4f25beea2af6eb9, State: Initialized, Role: FOLLOWER
I20260129 14:55:47.043561 8851 consensus_queue.cc:260] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.048246 8853 consensus_queue.cc:260] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.050200 8853 ts_tablet_manager.cc:1434] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260129 14:55:47.054402 8859 raft_consensus.cc:359] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.054778 8859 raft_consensus.cc:385] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:47.054868 8859 raft_consensus.cc:740] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0721a8132fc84354ae4b9e42fe387c67, State: Initialized, Role: FOLLOWER
I20260129 14:55:47.055594 8859 consensus_queue.cc:260] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.056625 8779 heartbeater.cc:499] Master 127.7.171.252:43411 was elected leader, sending a full tablet report...
I20260129 14:55:47.056620 8644 heartbeater.cc:499] Master 127.7.171.252:43411 was elected leader, sending a full tablet report...
I20260129 14:55:47.059036 8851 ts_tablet_manager.cc:1434] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Time spent starting tablet: real 0.031s user 0.010s sys 0.011s
I20260129 14:55:47.059995 8851 tablet_bootstrap.cc:492] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: Bootstrap starting.
I20260129 14:55:47.061673 8851 tablet_bootstrap.cc:654] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:47.063674 8859 ts_tablet_manager.cc:1434] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Time spent starting tablet: real 0.022s user 0.005s sys 0.006s
I20260129 14:55:47.064189 8859 tablet_bootstrap.cc:492] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Bootstrap starting.
I20260129 14:55:47.065522 8851 tablet_bootstrap.cc:492] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: No bootstrap required, opened a new log
I20260129 14:55:47.065681 8851 ts_tablet_manager.cc:1403] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260129 14:55:47.065863 8859 tablet_bootstrap.cc:654] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:47.066661 8851 raft_consensus.cc:359] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.066888 8851 raft_consensus.cc:385] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:47.066996 8851 raft_consensus.cc:740] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f177394443439dbabb0738a08b8fc3, State: Initialized, Role: FOLLOWER
I20260129 14:55:47.067216 8851 consensus_queue.cc:260] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.067673 8851 ts_tablet_manager.cc:1434] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260129 14:55:47.068181 8859 tablet_bootstrap.cc:492] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: No bootstrap required, opened a new log
I20260129 14:55:47.068347 8859 ts_tablet_manager.cc:1403] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260129 14:55:47.069424 8859 raft_consensus.cc:359] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.069597 8859 raft_consensus.cc:385] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:47.069680 8859 raft_consensus.cc:740] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0721a8132fc84354ae4b9e42fe387c67, State: Initialized, Role: FOLLOWER
I20260129 14:55:47.069895 8859 consensus_queue.cc:260] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.070369 8859 ts_tablet_manager.cc:1434] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260129 14:55:47.129009 8517 tablet.cc:2378] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260129 14:55:47.186844 8787 connection.cc:537] client connection to 127.7.171.252:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:47.186830 8668 connection.cc:537] client connection to 127.7.171.252:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:47.186893 8397 connection.cc:537] server connection from 127.0.0.1:43462 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:47.186841 8396 connection.cc:537] client connection to 127.7.171.252:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:47.188354 8787 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.7.171.252:43411)
W20260129 14:55:47.188378 8667 connection.cc:537] server connection from 127.0.0.1:40496 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:47.188736 8787 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.252:43411)
W20260129 14:55:47.188704 8533 connection.cc:537] server connection from 127.0.0.1:35096 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:47.188961 8533 connection.cc:537] client connection to 127.7.171.252:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:47.261969 8782 tablet.cc:2378] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260129 14:55:47.272926 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--webserver_interface=127.7.171.254
--webserver_port=33955
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:55:47.308367 8647 tablet.cc:2378] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260129 14:55:47.423804 8876 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8db7ebac21af437fbdf8e1bd7e00fca2)
I20260129 14:55:47.423985 8876 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:47.424754 8876 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:47.426846 8182 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:55:47.427009 8182 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:47.427099 8182 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c63d8c4ffa4047c9a8423771f4723d6e; no voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2
I20260129 14:55:47.427335 8876 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 14:55:47.594813 8875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:47.595104 8875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:47.595217 8875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:47.603294 8875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:55:47.603396 8875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:47.603495 8875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:55:47.603565 8875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:55:47.612912 8875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.171.254
--webserver_port=33955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:47.614272 8875 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:47.615947 8875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:55:47.623102 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:47.623200 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:47.623761 8887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:47.624325 8875 server_base.cc:1060] running on GCE node
I20260129 14:55:47.624963 8875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:47.626209 8875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:47.627398 8875 hybrid_clock.cc:648] HybridClock initialized: now 1769698547627343 us; error 51 us; skew 500 ppm
I20260129 14:55:47.627692 8875 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:47.629601 8875 webserver.cc:492] Webserver started at http://127.7.171.254:33955/ using document root <none> and password file <none>
I20260129 14:55:47.630264 8875 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:47.630383 8875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:47.635404 8875 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260129 14:55:47.638134 8893 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:47.639370 8875 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260129 14:55:47.639528 8875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
uuid: "5cde2b69b78d4e8491572266f7019c6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:47.640204 8875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:47.667698 8898 raft_consensus.cc:493] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:47.667937 8898 raft_consensus.cc:515] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.669261 8875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:47.669694 8898 leader_election.cc:290] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151), 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:47.670339 8875 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:47.670684 8875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:47.676615 8899 raft_consensus.cc:493] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:47.677500 8899 raft_consensus.cc:515] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.696576 8899 leader_election.cc:290] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151), e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:47.700493 8599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5bc5c45b2a4314b92c797aac0ca371" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0721a8132fc84354ae4b9e42fe387c67" is_pre_election: true
I20260129 14:55:47.701072 8599 raft_consensus.cc:2468] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 0.
I20260129 14:55:47.704113 8668 leader_election.cc:304] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0721a8132fc84354ae4b9e42fe387c67, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:55:47.704671 8898 raft_consensus.cc:2804] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 14:55:47.709012 8898 raft_consensus.cc:493] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:55:47.709609 8462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5bc5c45b2a4314b92c797aac0ca371" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9" is_pre_election: true
I20260129 14:55:47.710060 8462 raft_consensus.cc:2468] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 0.
I20260129 14:55:47.709210 8734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7c50ee4a0464ec4a03a6a30e4cb74df" candidate_uuid: "0721a8132fc84354ae4b9e42fe387c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f177394443439dbabb0738a08b8fc3" is_pre_election: true
I20260129 14:55:47.710749 8734 raft_consensus.cc:2468] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0721a8132fc84354ae4b9e42fe387c67 in term 0.
I20260129 14:55:47.709134 8898 raft_consensus.cc:3060] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:47.711256 8533 leader_election.cc:304] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0721a8132fc84354ae4b9e42fe387c67, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:55:47.711750 8899 raft_consensus.cc:2804] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 14:55:47.711912 8899 raft_consensus.cc:493] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:55:47.712018 8899 raft_consensus.cc:3060] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:47.714051 8462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7c50ee4a0464ec4a03a6a30e4cb74df" candidate_uuid: "0721a8132fc84354ae4b9e42fe387c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9" is_pre_election: true
I20260129 14:55:47.714274 8462 raft_consensus.cc:2468] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0721a8132fc84354ae4b9e42fe387c67 in term 0.
I20260129 14:55:47.714740 8898 raft_consensus.cc:515] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.714984 8899 raft_consensus.cc:515] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.715602 8898 leader_election.cc:290] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151), 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:47.716184 8734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7c50ee4a0464ec4a03a6a30e4cb74df" candidate_uuid: "0721a8132fc84354ae4b9e42fe387c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f177394443439dbabb0738a08b8fc3"
I20260129 14:55:47.716259 8599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5bc5c45b2a4314b92c797aac0ca371" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0721a8132fc84354ae4b9e42fe387c67"
I20260129 14:55:47.716398 8734 raft_consensus.cc:3060] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:47.716464 8599 raft_consensus.cc:3060] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:47.716470 8462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7c50ee4a0464ec4a03a6a30e4cb74df" candidate_uuid: "0721a8132fc84354ae4b9e42fe387c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9"
I20260129 14:55:47.716387 8461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5bc5c45b2a4314b92c797aac0ca371" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9"
I20260129 14:55:47.716656 8462 raft_consensus.cc:3060] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:47.716674 8461 raft_consensus.cc:3060] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:47.719221 8734 raft_consensus.cc:2468] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0721a8132fc84354ae4b9e42fe387c67 in term 1.
I20260129 14:55:47.719369 8599 raft_consensus.cc:2468] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 1.
I20260129 14:55:47.719612 8533 leader_election.cc:304] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0721a8132fc84354ae4b9e42fe387c67, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:55:47.719786 8461 raft_consensus.cc:2468] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 1.
I20260129 14:55:47.719810 8668 leader_election.cc:304] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0721a8132fc84354ae4b9e42fe387c67, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:55:47.719848 8462 raft_consensus.cc:2468] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0721a8132fc84354ae4b9e42fe387c67 in term 1.
I20260129 14:55:47.716279 8899 leader_election.cc:290] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [CANDIDATE]: Term 1 election: Requested vote from peers 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151), e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:47.720517 8899 raft_consensus.cc:2804] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 14:55:47.720703 8899 raft_consensus.cc:697] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 1 LEADER]: Becoming Leader. State: Replica: 0721a8132fc84354ae4b9e42fe387c67, State: Running, Role: LEADER
I20260129 14:55:47.721249 8899 consensus_queue.cc:237] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } }
I20260129 14:55:47.720117 8898 raft_consensus.cc:2804] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 14:55:47.723903 8898 raft_consensus.cc:697] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 1 LEADER]: Becoming Leader. State: Replica: e6f177394443439dbabb0738a08b8fc3, State: Running, Role: LEADER
I20260129 14:55:47.724426 8898 consensus_queue.cc:237] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:55:47.741555 8875 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.254:38695
I20260129 14:55:47.741592 8957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.254:38695 every 8 connection(s)
I20260129 14:55:47.743006 8875 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
I20260129 14:55:47.749300 8958 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:55:47.751224 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 8875
I20260129 14:55:47.753091 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap starting.
I20260129 14:55:47.762513 8958 log.cc:826] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:47.773533 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 14:55:47.774026 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap complete.
I20260129 14:55:47.778738 8958 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:47.779284 8958 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Initialized, Role: FOLLOWER
I20260129 14:55:47.779878 8958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:47.780659 8962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:47.780896 8962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:47.781486 8958 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:55:47.791285 8976 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:47.791424 8976 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: loading cluster ID for follower catalog manager: success
I20260129 14:55:47.797219 8976 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: acquiring CA information for follower catalog manager: success
I20260129 14:55:47.802135 8976 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:55:47.822621 8875 master_runner.cc:186] Error getting master registration for 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111),
I20260129 14:55:47.825932 8875 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:55:48.059882 8508 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:48.062397 8923 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:40099; Asking this server to re-register.
I20260129 14:55:48.063107 8508 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:48.064576 8923 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:48.126741 8981 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8db7ebac21af437fbdf8e1bd7e00fca2)
I20260129 14:55:48.126909 8981 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:48.127470 8981 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:48.130633 8182 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:48.135736 8933 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e" is_pre_election: true
I20260129 14:55:48.136298 8933 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63d8c4ffa4047c9a8423771f4723d6e in term 1.
I20260129 14:55:48.136765 8182 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2
I20260129 14:55:48.137048 8981 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 14:55:48.137173 8981 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8db7ebac21af437fbdf8e1bd7e00fca2)
I20260129 14:55:48.137274 8981 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 2 FOLLOWER]: Advancing to term 3
I20260129 14:55:48.139218 8981 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:48.139780 8981 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 election: Requested vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:55:48.140085 8933 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e"
I20260129 14:55:48.140283 8933 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 1 FOLLOWER]: Advancing to term 3
W20260129 14:55:48.141005 8182 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:48.145251 8933 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63d8c4ffa4047c9a8423771f4723d6e in term 3.
I20260129 14:55:48.145630 8182 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2
I20260129 14:55:48.145922 8981 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Leader election won for term 3
I20260129 14:55:48.146548 8981 raft_consensus.cc:697] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 LEADER]: Becoming Leader. State: Replica: c63d8c4ffa4047c9a8423771f4723d6e, State: Running, Role: LEADER
I20260129 14:55:48.146951 8981 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:48.148433 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63d8c4ffa4047c9a8423771f4723d6e. Latest consensus state: current_term: 3 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:48.148669 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:48.149384 8987 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:55:48.152438 8987 catalog_manager.cc:679] Loaded metadata for table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]
I20260129 14:55:48.152956 8987 catalog_manager.cc:679] Loaded metadata for table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]
I20260129 14:55:48.154712 8987 tablet_loader.cc:96] loaded metadata for tablet 7b5bc5c45b2a4314b92c797aac0ca371 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed])
I20260129 14:55:48.155102 8987 tablet_loader.cc:96] loaded metadata for tablet d7c50ee4a0464ec4a03a6a30e4cb74df (table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc])
I20260129 14:55:48.155350 8987 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:55:48.156278 8987 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:48.156352 8987 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:55:48.157668 8987 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:55:48.158656 8987 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Loaded TSK: 0
I20260129 14:55:48.159603 8987 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:55:48.161108 8987 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:55:48.162161 8987 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:55:48.209172 8195 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_b6e29b73e03042138b5a216f08dce0c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.210178 8192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.209698 8193 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.210661 8197 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.210427 8189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_17c52877671f4e30bcf4f682122e181e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.209546 8196 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.210500 8190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_60640f05a979404380083e385f1e6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.211043 8188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52774:
table { table_name: "default.Table_c2b80faabf21435d991d592f638440aa" } new_table_name: "default.Table_2ec3463f89a9421cac3c15af609890f3" modify_external_catalogs: true
I20260129 14:55:48.210062 8191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:48.209877 8194 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52774:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260129 14:55:48.219730 8374 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_c2b80faabf21435d991d592f638440aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2b80faabf21435d991d592f638440aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2b80faabf21435d991d592f638440aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_c2b80faabf21435d991d592f638440aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2b80faabf21435d991d592f638440aa\",\"kudu.table_id\":\"908853f113ee48df891ae3a9db09c1ed\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_95ece788510d4811b741bfafc484dd89, message={"server":"","servicePrincipal":"","db":"default","table":"table_95ece788510d4811b741bfafc484dd89","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_95ece788510d4811b741bfafc484dd89\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_95ece788510d4811b741bfafc484dd89\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_95ece788510d4811b741bfafc484dd89\",\"kudu.table_id\":\"66667881263c4bb0a181c0c7576354f7\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fb7a08e6dcd google::LogMessage::Fail() at ??:0
@ 0x7fb7a08eab93 google::LogMessage::SendToLog() at ??:0
I20260129 14:55:48.234216 8780 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:48.234392 8780 heartbeater.cc:499] Master 127.7.171.254:38695 was elected leader, sending a full tablet report...
I20260129 14:55:48.236006 8923 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:33363; Asking this server to re-register.
I20260129 14:55:48.236752 8780 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:48.237952 8923 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
@ 0x7fb7a08e67cc google::LogMessage::Flush() at ??:0
@ 0x7fb7a08e6a11 google::LogMessage::~LogMessage() at ??:0
I20260129 14:55:48.243844 8645 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:48.243975 8645 heartbeater.cc:499] Master 127.7.171.254:38695 was elected leader, sending a full tablet report...
I20260129 14:55:48.245234 8923 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:51143; Asking this server to re-register.
I20260129 14:55:48.245798 8645 heartbeater.cc:461] Registering TS with master...
@ 0x7fb7b72dd7f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260129 14:55:48.246831 8923 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
@ 0x7fb7b72dc0a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb7b72e3d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fb7b72e3b99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fb7a365bf99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb7a6c686db start_thread at ??:0
@ 0x7fb79f79771f clone at ??:0
W20260129 14:55:48.422636 8786 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.171.253:34637)
W20260129 14:55:48.423569 8786 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.7.171.253:34637)
W20260129 14:55:48.423875 8786 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.253:34637)
I20260129 14:55:48.746986 9001 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:48.747134 9001 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:48.749042 9001 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:48.749904 8894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:48.751331 8894 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:48.751513 8895 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:48.751641 8895 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:48.751986 9001 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 14:55:48.773736 9005 consensus_queue.cc:1048] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:55:48.782981 9009 consensus_queue.cc:1048] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:55:48.846966 9015 consensus_queue.cc:1048] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:55:48.849504 9019 consensus_queue.cc:1048] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:55:49.403115 9029 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:49.403288 9029 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:49.404031 9029 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:49.406051 8894 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:49.406215 8895 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:49.406356 8895 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:49.406600 9029 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 14:55:50.304605 7855 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu pid 8875
I20260129 14:55:50.311043 8953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260129 14:55:50.416365 9048 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:50.416584 9048 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:50.416682 7855 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu with pid 8875
I20260129 14:55:51.260738 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--webserver_interface=127.7.171.254
--webserver_port=33955
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:55:51.571769 9059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:51.572067 9059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:51.572163 9059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:51.579886 9059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:55:51.579990 9059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:51.580071 9059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:55:51.580125 9059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:55:51.589726 9059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.171.254
--webserver_port=33955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:51.591197 9059 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:51.592962 9059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:55:51.600185 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:51.600226 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:51.600579 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:51.601716 9059 server_base.cc:1060] running on GCE node
I20260129 14:55:51.602371 9059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:51.603539 9059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:51.604727 9059 hybrid_clock.cc:648] HybridClock initialized: now 1769698551604688 us; error 35 us; skew 500 ppm
I20260129 14:55:51.605032 9059 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:51.607160 9059 webserver.cc:492] Webserver started at http://127.7.171.254:33955/ using document root <none> and password file <none>
I20260129 14:55:51.607827 9059 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:51.607930 9059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:51.612680 9059 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260129 14:55:51.615402 9076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:51.616796 9059 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260129 14:55:51.616940 9059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
uuid: "5cde2b69b78d4e8491572266f7019c6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:51.617580 9059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:51.633581 9059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:51.634403 9059 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:51.634636 9059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:51.654467 9059 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.254:38695
I20260129 14:55:51.654495 9129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.254:38695 every 8 connection(s)
I20260129 14:55:51.656200 9059 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
I20260129 14:55:51.657366 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 9059
I20260129 14:55:51.664606 9130 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:55:51.668759 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap starting.
I20260129 14:55:51.677214 9130 log.cc:826] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:51.684551 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 14:55:51.684999 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap complete.
I20260129 14:55:51.689694 9130 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:51.690374 9130 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Initialized, Role: FOLLOWER
I20260129 14:55:51.690912 9130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:51.691762 9134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:51.692107 9134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:51.692822 9130 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:55:51.706225 9148 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:51.706415 9148 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: loading cluster ID for follower catalog manager: success
I20260129 14:55:51.710125 9148 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: acquiring CA information for follower catalog manager: success
I20260129 14:55:51.712641 9148 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:55:51.720546 9059 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111),
I20260129 14:55:51.725484 9059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:55:51.872987 8780 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:51.876116 9093 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:58775; Asking this server to re-register.
I20260129 14:55:51.876839 8780 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:51.878538 9093 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:51.926764 8508 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:51.927960 9093 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:54681; Asking this server to re-register.
I20260129 14:55:51.928571 8508 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:51.929648 9093 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:52.152633 8645 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:55:52.153901 9093 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:57663; Asking this server to re-register.
I20260129 14:55:52.154415 8645 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:52.155398 9093 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:52.290990 9160 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:52.291285 9160 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:52.293206 9160 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:52.294214 9077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:52.303725 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
W20260129 14:55:52.304047 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:52.304205 9077 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:52.304606 9160 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 14:55:52.942461 9170 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:52.942629 9170 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:52.943224 9170 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:52.945258 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:52.945498 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:52.945665 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:52.945887 9170 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 14:55:53.705564 9181 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:53.705727 9181 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:53.706362 9181 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:53.708318 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:53.708539 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:53.708667 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:53.708885 9181 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 14:55:54.239897 9192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:54.240130 9192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:54.240782 9192 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:54.242774 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:54.242965 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:54.243121 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:54.243343 9192 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 14:55:54.514200 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--webserver_interface=127.7.171.252
--webserver_port=37043
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:55:54.849187 9200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:54.849557 9200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:54.849674 9200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:54.858407 9200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:55:54.858530 9200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:54.858633 9200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:55:54.858703 9200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:55:54.869402 9200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.171.252
--webserver_port=37043
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:54.871119 9200 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:54.873950 9200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:55:54.889035 9213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:54.889173 9214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:54.889592 9216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:54.890373 9200 server_base.cc:1060] running on GCE node
I20260129 14:55:54.891207 9200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:54.892755 9200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:54.893959 9200 hybrid_clock.cc:648] HybridClock initialized: now 1769698554893931 us; error 31 us; skew 500 ppm
I20260129 14:55:54.894294 9200 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:54.896631 9200 webserver.cc:492] Webserver started at http://127.7.171.252:37043/ using document root <none> and password file <none>
I20260129 14:55:54.897356 9200 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:54.897493 9200 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:54.903183 9200 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 14:55:54.906153 9222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:54.907557 9200 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260129 14:55:54.907753 9200 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
format_stamp: "Formatted at 2026-01-29 14:55:44 on dist-test-slave-n0m6"
I20260129 14:55:54.908500 9200 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:54.924927 9200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:54.925832 9200 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:54.926103 9200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:54.946460 9200 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.252:43411
I20260129 14:55:54.946482 9273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.252:43411 every 8 connection(s)
I20260129 14:55:54.948211 9200 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
I20260129 14:55:54.953011 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 9200
I20260129 14:55:54.955971 9274 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:55:54.959364 9274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap starting.
I20260129 14:55:54.968075 9274 log.cc:826] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:54.978647 9274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=7 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 14:55:54.979231 9274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap complete.
I20260129 14:55:54.984449 9274 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:54.985132 9274 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db7ebac21af437fbdf8e1bd7e00fca2, State: Initialized, Role: FOLLOWER
I20260129 14:55:54.985798 9274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:54.986629 9278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:54.986896 9278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:54.987538 9274 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:55:55.001194 9292 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:55.001392 9292 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: loading cluster ID for follower catalog manager: success
I20260129 14:55:55.005599 9292 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: acquiring CA information for follower catalog manager: success
I20260129 14:55:55.008554 9292 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:55:55.025238 9200 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111),
I20260129 14:55:55.030310 9200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:55:55.162931 8644 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:55:55.163146 8644 heartbeater.cc:499] Master 127.7.171.252:43411 was elected leader, sending a full tablet report...
I20260129 14:55:55.166242 9239 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:40569; Asking this server to re-register.
I20260129 14:55:55.167248 8644 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:55.169449 9239 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:55.395253 8779 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:55:55.395474 8779 heartbeater.cc:499] Master 127.7.171.252:43411 was elected leader, sending a full tablet report...
I20260129 14:55:55.396848 9239 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:50547; Asking this server to re-register.
I20260129 14:55:55.397137 8507 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:55:55.397329 8507 heartbeater.cc:499] Master 127.7.171.252:43411 was elected leader, sending a full tablet report...
I20260129 14:55:55.397353 8779 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:55.398490 9239 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:55701; Asking this server to re-register.
I20260129 14:55:55.398864 9238 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:55.399021 8507 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:55.400017 9238 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:55.540297 9305 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:55.540617 9305 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:55.542487 9305 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
I20260129 14:55:55.543243 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e" is_pre_election: true
W20260129 14:55:55.543390 9223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:55.543776 9105 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8db7ebac21af437fbdf8e1bd7e00fca2 in current term 3: Already voted for candidate c63d8c4ffa4047c9a8423771f4723d6e in this term.
W20260129 14:55:55.544787 9223 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:55.544915 9223 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:55.545212 9305 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 2 FOLLOWER]: Advancing to term 3
I20260129 14:55:55.548712 9305 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 14:55:56.176728 9309 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:56.176888 9309 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:56.177398 9309 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
I20260129 14:55:56.177811 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e" is_pre_election: true
I20260129 14:55:56.178035 9105 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db7ebac21af437fbdf8e1bd7e00fca2 in term 3.
I20260129 14:55:56.178439 9224 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters:
I20260129 14:55:56.178704 9309 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 14:55:56.178819 9309 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:55:56.178926 9309 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 3 FOLLOWER]: Advancing to term 4
W20260129 14:55:56.179741 9223 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:56.181022 9309 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:56.181644 9309 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 4 election: Requested vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
I20260129 14:55:56.181982 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e"
I20260129 14:55:56.182179 9105 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 3 FOLLOWER]: Advancing to term 4
W20260129 14:55:56.182684 9223 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:55:56.185616 9105 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db7ebac21af437fbdf8e1bd7e00fca2 in term 4.
I20260129 14:55:56.185959 9224 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:56.186225 9309 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 14:55:56.186591 9309 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 4 LEADER]: Becoming Leader. State: Replica: 8db7ebac21af437fbdf8e1bd7e00fca2, State: Running, Role: LEADER
I20260129 14:55:56.187016 9309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:56.189391 9312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db7ebac21af437fbdf8e1bd7e00fca2. Latest consensus state: current_term: 4 leader_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:56.189576 9312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:56.190092 9314 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:55:56.192503 9314 catalog_manager.cc:679] Loaded metadata for table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]
I20260129 14:55:56.192898 9314 catalog_manager.cc:679] Loaded metadata for table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]
I20260129 14:55:56.194288 9314 tablet_loader.cc:96] loaded metadata for tablet 7b5bc5c45b2a4314b92c797aac0ca371 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed])
I20260129 14:55:56.194622 9314 tablet_loader.cc:96] loaded metadata for tablet d7c50ee4a0464ec4a03a6a30e4cb74df (table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc])
I20260129 14:55:56.194844 9314 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:55:56.195587 9314 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:56.195659 9314 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:55:56.196775 9314 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:55:56.197403 9314 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Loaded TSK: 0
I20260129 14:55:56.198112 9314 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:55:56.198848 9314 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:55:56.199478 9314 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:55:56.243692 9238 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_ffffe38dc46a4a97ba5f4b081cbcc822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:56.243790 9237 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:56.243734 9239 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:56.243963 9236 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:56.245190 9235 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:56.249603 9234 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_b6e29b73e03042138b5a216f08dce0c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260129 14:55:56.250469 9291 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_c2b80faabf21435d991d592f638440aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2b80faabf21435d991d592f638440aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2b80faabf21435d991d592f638440aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_c2b80faabf21435d991d592f638440aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2b80faabf21435d991d592f638440aa\",\"kudu.table_id\":\"908853f113ee48df891ae3a9db09c1ed\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_95ece788510d4811b741bfafc484dd89, message={"server":"","servicePrincipal":"","db":"default","table":"table_95ece788510d4811b741bfafc484dd89","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_95ece788510d4811b741bfafc484dd89\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_95ece788510d4811b741bfafc484dd89\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_95ece788510d4811b741bfafc484dd89\",\"kudu.table_id\":\"66667881263c4bb0a181c0c7576354f7\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260129 14:55:56.250063 9232 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:56.250332 9233 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:56.252142 9231 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_60640f05a979404380083e385f1e6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f4271ccadcd google::LogMessage::Fail() at ??:0
I20260129 14:55:56.252616 9230 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36650:
name: "default.Table_17c52877671f4e30bcf4f682122e181e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f4271cceb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4271cca7cc google::LogMessage::Flush() at ??:0
@ 0x7f4271ccaa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f42886c17f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f42886c00a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f42886c7d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f42886c7b99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f4274a3ff99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f427804c6db start_thread at ??:0
@ 0x7f4270b7b71f clone at ??:0
W20260129 14:55:56.438373 8396 connection.cc:537] client connection to 127.7.171.252:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:56.439119 8668 connection.cc:537] client connection to 127.7.171.252:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:55:56.439182 8787 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.252:43411)
W20260129 14:55:56.439412 8787 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.171.252:43411)
W20260129 14:55:56.440393 8787 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.7.171.252:43411)
I20260129 14:55:56.728044 9059 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111),
I20260129 14:55:56.728274 9059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:55:57.050483 9335 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:57.050729 9335 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:57.051496 9335 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:57.053332 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:55:57.053509 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:57.053639 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:57.053859 9335 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 14:55:58.209187 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.253:34637
--webserver_interface=127.7.171.253
--webserver_port=41757
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:55:58.512290 9340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:55:58.512569 9340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:55:58.512697 9340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:55:58.520769 9340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:55:58.520850 9340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:55:58.520922 9340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:55:58.520988 9340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:55:58.529881 9340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.253:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.171.253
--webserver_port=41757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:55:58.531394 9340 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:55:58.533102 9340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:55:58.540380 9348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:58.540414 9347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:55:58.541080 9350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:55:58.541837 9340 server_base.cc:1060] running on GCE node
I20260129 14:55:58.542444 9340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:55:58.543634 9340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:55:58.544828 9340 hybrid_clock.cc:648] HybridClock initialized: now 1769698558544791 us; error 51 us; skew 500 ppm
I20260129 14:55:58.545112 9340 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:55:58.547000 9340 webserver.cc:492] Webserver started at http://127.7.171.253:41757/ using document root <none> and password file <none>
I20260129 14:55:58.547960 9340 fs_manager.cc:362] Metadata directory not provided
I20260129 14:55:58.548101 9340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:55:58.552953 9340 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260129 14:55:58.555373 9356 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:55:58.556624 9340 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 14:55:58.556773 9340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
uuid: "c63d8c4ffa4047c9a8423771f4723d6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:55:58.557428 9340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:55:58.573179 9340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:55:58.573987 9340 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:55:58.574209 9340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:55:58.591777 9340 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.253:34637
I20260129 14:55:58.591804 9407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.253:34637 every 8 connection(s)
I20260129 14:55:58.593159 9340 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
I20260129 14:55:58.594784 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 9340
I20260129 14:55:58.600162 9408 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:55:58.603744 9408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Bootstrap starting.
I20260129 14:55:58.610986 9408 log.cc:826] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:55:58.620995 9408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=0} inserts{seen=7 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260129 14:55:58.621520 9408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Bootstrap complete.
I20260129 14:55:58.626547 9408 raft_consensus.cc:359] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:58.627426 9408 raft_consensus.cc:740] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63d8c4ffa4047c9a8423771f4723d6e, State: Initialized, Role: FOLLOWER
I20260129 14:55:58.628015 9408 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:58.628736 9412 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:58.628937 9412 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:58.629504 9408 sys_catalog.cc:565] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:55:58.640452 9426 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:58.640609 9426 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: loading cluster ID for follower catalog manager: success
I20260129 14:55:58.644503 9426 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: acquiring CA information for follower catalog manager: success
I20260129 14:55:58.647234 9426 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:55:58.660286 9340 master_runner.cc:186] Error getting master registration for 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111),
I20260129 14:55:58.665825 9340 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:55:58.670967 8646 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:55:58.674232 9373 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:34715; Asking this server to re-register.
I20260129 14:55:58.674911 8646 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:58.676283 9373 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:55:58.821672 9431 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:58.821853 9431 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:58.822633 9431 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:55:58.823984 9078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260129 14:55:58.825752 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:58.829926 9383 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" is_pre_election: true
I20260129 14:55:58.830619 9383 raft_consensus.cc:2410] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5cde2b69b78d4e8491572266f7019c6e for term 5 because replica has last-logged OpId of term: 3 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260129 14:55:58.831271 9077 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:55:58.831594 9431 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 14:55:58.906280 8512 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:55:58.906702 8781 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:55:58.907771 9373 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:35473; Asking this server to re-register.
I20260129 14:55:58.908111 9372 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:48473; Asking this server to re-register.
I20260129 14:55:58.908357 8512 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:58.909169 8781 heartbeater.cc:461] Registering TS with master...
I20260129 14:55:58.909421 9372 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:58.910276 9372 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:55:59.145828 9437 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:59.146054 9437 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:59.147743 9437 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:55:59.148252 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e" is_pre_election: true
W20260129 14:55:59.148468 9358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:59.148576 9105 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c63d8c4ffa4047c9a8423771f4723d6e in current term 4: Already voted for candidate 8db7ebac21af437fbdf8e1bd7e00fca2 in this term.
W20260129 14:55:59.149717 9358 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:59.149834 9358 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c63d8c4ffa4047c9a8423771f4723d6e; no voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2
I20260129 14:55:59.150135 9437 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 3 FOLLOWER]: Advancing to term 4
I20260129 14:55:59.153764 9437 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 14:55:59.784896 9444 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:59.785072 9444 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:59.785713 9444 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:55:59.786222 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 5 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e" is_pre_election: true
I20260129 14:55:59.786432 9105 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63d8c4ffa4047c9a8423771f4723d6e in term 4.
I20260129 14:55:59.787005 9358 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e; no voters:
I20260129 14:55:59.787282 9444 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 14:55:59.787407 9444 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:55:59.787520 9444 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 4 FOLLOWER]: Advancing to term 5
W20260129 14:55:59.787761 9358 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:59.789556 9444 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:59.789987 9444 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 5 election: Requested vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:55:59.790390 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 5 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e"
I20260129 14:55:59.790583 9105 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 4 FOLLOWER]: Advancing to term 5
W20260129 14:55:59.791237 9358 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:55:59.792347 9105 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63d8c4ffa4047c9a8423771f4723d6e in term 5.
I20260129 14:55:59.792667 9358 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2
I20260129 14:55:59.792912 9444 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Leader election won for term 5
I20260129 14:55:59.793241 9444 raft_consensus.cc:697] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 LEADER]: Becoming Leader. State: Replica: c63d8c4ffa4047c9a8423771f4723d6e, State: Running, Role: LEADER
I20260129 14:55:59.793623 9444 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.10, Last appended by leader: 10, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:55:59.796031 9447 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63d8c4ffa4047c9a8423771f4723d6e. Latest consensus state: current_term: 5 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:59.796293 9447 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:59.796962 9449 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:55:59.799126 9449 catalog_manager.cc:679] Loaded metadata for table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]
I20260129 14:55:59.799468 9449 catalog_manager.cc:679] Loaded metadata for table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]
I20260129 14:55:59.800997 9449 tablet_loader.cc:96] loaded metadata for tablet 7b5bc5c45b2a4314b92c797aac0ca371 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed])
I20260129 14:55:59.801352 9449 tablet_loader.cc:96] loaded metadata for tablet d7c50ee4a0464ec4a03a6a30e4cb74df (table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc])
I20260129 14:55:59.801599 9449 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:55:59.802421 9449 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:55:59.802510 9449 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:55:59.803470 9449 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:55:59.804176 9449 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Loaded TSK: 0
I20260129 14:55:59.804958 9449 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:55:59.805713 9449 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:55:59.806435 9449 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:55:59.818671 9372 master_service.cc:954] ReplaceTablet: received request to replace tablet 7b5bc5c45b2a4314b92c797aac0ca371 from {username='slave'} at 127.0.0.1:59916
I20260129 14:55:59.819100 9373 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_b6e29b73e03042138b5a216f08dce0c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.819427 9369 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.820091 9370 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.820443 9371 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.820914 9368 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_ffffe38dc46a4a97ba5f4b081cbcc822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.821341 9364 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_60640f05a979404380083e385f1e6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.821727 9367 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.822185 9365 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.822750 9366 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260129 14:55:59.825495 9425 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_c2b80faabf21435d991d592f638440aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2b80faabf21435d991d592f638440aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2b80faabf21435d991d592f638440aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_c2b80faabf21435d991d592f638440aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2b80faabf21435d991d592f638440aa\",\"kudu.table_id\":\"908853f113ee48df891ae3a9db09c1ed\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_95ece788510d4811b741bfafc484dd89, message={"server":"","servicePrincipal":"","db":"default","table":"table_95ece788510d4811b741bfafc484dd89","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_95ece788510d4811b741bfafc484dd89\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_95ece788510d4811b741bfafc484dd89\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_95ece788510d4811b741bfafc484dd89\",\"kudu.table_id\":\"66667881263c4bb0a181c0c7576354f7\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
W20260129 14:55:59.826148 9358 consensus_peers.cc:597] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e -> Peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Couldn't send request to peer 8db7ebac21af437fbdf8e1bd7e00fca2. Status: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 14:55:59.826411 9105 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Refusing update from remote peer c63d8c4ffa4047c9a8423771f4723d6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 12. (index mismatch)
I20260129 14:55:59.827003 9444 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f5935c62dcd google::LogMessage::Fail() at ??:0
@ 0x7f5935c66b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5935c627cc google::LogMessage::Flush() at ??:0
@ 0x7f5935c62a11 google::LogMessage::~LogMessage() at ??:0
I20260129 14:55:59.836606 9456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:59.836879 9456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:59.836880 9452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63d8c4ffa4047c9a8423771f4723d6e. Latest consensus state: current_term: 5 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:59.837203 9452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f594c6597f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260129 14:55:59.840432 9456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:59.840629 9456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:59.840814 9456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:59.840967 9456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:55:59.842304 9372 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Sending DeleteTablet for 3 replicas of tablet 7b5bc5c45b2a4314b92c797aac0ca371
I20260129 14:55:59.844197 9372 catalog_manager.cc:6724] ReplaceTablet: tablet 7b5bc5c45b2a4314b92c797aac0ca371 deleted and replaced by tablet 563c97415304403c8a50ff956fb54363
I20260129 14:55:59.844106 9450 mvcc.cc:204] Tried to move back new op lower bound from 7248685301038534656 to 7248685253213122560. Current Snapshot: MvccSnapshot[applied={T|T < 7248685301038534656 or (T in {7248685301038534656})}]
I20260129 14:55:59.847388 9447 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:59.847534 9447 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:59.850162 9453 mvcc.cc:204] Tried to move back new op lower bound from 7248685301038534656 to 7248685247212048384. Current Snapshot: MvccSnapshot[applied={T|T < 7248685301038534656 or (T in {7248685301038534656})}]
@ 0x7f594c6580a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260129 14:55:59.850363 9372 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59916:
name: "default.Table_17c52877671f4e30bcf4f682122e181e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:55:59.847018 9461 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:55:59.853554 9461 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:55:59.860209 8441 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5bc5c45b2a4314b92c797aac0ca371 with delete_type TABLET_DATA_DELETED (replaced by tablet 563c97415304403c8a50ff956fb54363) from {username='slave'} at 127.0.0.1:46128
I20260129 14:55:59.865473 9469 tablet_replica.cc:333] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: stopping tablet replica
I20260129 14:55:59.865926 9469 raft_consensus.cc:2243] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 14:55:59.866304 9469 raft_consensus.cc:2272] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 14:55:59.867208 8714 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5bc5c45b2a4314b92c797aac0ca371 with delete_type TABLET_DATA_DELETED (replaced by tablet 563c97415304403c8a50ff956fb54363) from {username='slave'} at 127.0.0.1:49544
I20260129 14:55:59.867985 9471 tablet_replica.cc:333] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: stopping tablet replica
I20260129 14:55:59.868096 8578 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5bc5c45b2a4314b92c797aac0ca371 with delete_type TABLET_DATA_DELETED (replaced by tablet 563c97415304403c8a50ff956fb54363) from {username='slave'} at 127.0.0.1:48964
I20260129 14:55:59.868310 9471 raft_consensus.cc:2243] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 1 LEADER]: Raft consensus shutting down.
I20260129 14:55:59.868768 9471 raft_consensus.cc:2272] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 14:55:59.869733 9471 ts_tablet_manager.cc:1916] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 14:55:59.872466 9469 ts_tablet_manager.cc:1916] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 14:55:59.872694 9472 tablet_replica.cc:333] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: stopping tablet replica
I20260129 14:55:59.872960 9472 raft_consensus.cc:2243] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 14:55:59.873149 9472 raft_consensus.cc:2272] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 14:55:59.874007 9472 ts_tablet_manager.cc:1916] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 14:55:59.875164 9471 ts_tablet_manager.cc:1929] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 14:55:59.875286 9471 log.cc:1199] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Deleting WAL directory at /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal/wals/7b5bc5c45b2a4314b92c797aac0ca371
I20260129 14:55:59.875741 9471 ts_tablet_manager.cc:1950] T 7b5bc5c45b2a4314b92c797aac0ca371 P e6f177394443439dbabb0738a08b8fc3: Deleting consensus metadata
I20260129 14:55:59.878039 9469 ts_tablet_manager.cc:1929] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 14:55:59.878129 9469 log.cc:1199] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Deleting WAL directory at /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal/wals/7b5bc5c45b2a4314b92c797aac0ca371
I20260129 14:55:59.878476 9469 ts_tablet_manager.cc:1950] T 7b5bc5c45b2a4314b92c797aac0ca371 P 824b258cd009461bb4f25beea2af6eb9: Deleting consensus metadata
I20260129 14:55:59.879240 9472 ts_tablet_manager.cc:1929] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 14:55:59.879364 9472 log.cc:1199] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Deleting WAL directory at /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal/wals/7b5bc5c45b2a4314b92c797aac0ca371
I20260129 14:55:59.879799 9472 ts_tablet_manager.cc:1950] T 7b5bc5c45b2a4314b92c797aac0ca371 P 0721a8132fc84354ae4b9e42fe387c67: Deleting consensus metadata
@ 0x7f594c65fd1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260129 14:55:59.881372 9358 catalog_manager.cc:4985] TS e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395): tablet 7b5bc5c45b2a4314b92c797aac0ca371 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]) successfully deleted
I20260129 14:55:59.881999 9358 catalog_manager.cc:4985] TS 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263): tablet 7b5bc5c45b2a4314b92c797aac0ca371 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]) successfully deleted
I20260129 14:55:59.882241 9359 catalog_manager.cc:4985] TS 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151): tablet 7b5bc5c45b2a4314b92c797aac0ca371 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]) successfully deleted
I20260129 14:55:59.882301 8578 tablet_service.cc:1505] Processing CreateTablet for tablet 563c97415304403c8a50ff956fb54363 (DEFAULT_TABLE table=default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:59.882378 8714 tablet_service.cc:1505] Processing CreateTablet for tablet 563c97415304403c8a50ff956fb54363 (DEFAULT_TABLE table=default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:59.882701 8578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563c97415304403c8a50ff956fb54363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:59.882730 8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563c97415304403c8a50ff956fb54363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:55:59.883013 8441 tablet_service.cc:1505] Processing CreateTablet for tablet 563c97415304403c8a50ff956fb54363 (DEFAULT_TABLE table=default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:55:59.883428 8441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563c97415304403c8a50ff956fb54363. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f594c65fb99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f59389d7f99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f593bfe46db start_thread at ??:0
@ 0x7f5934b1371f clone at ??:0
I20260129 14:55:59.895766 9477 tablet_bootstrap.cc:492] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9: Bootstrap starting.
I20260129 14:55:59.896739 9476 tablet_bootstrap.cc:492] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67: Bootstrap starting.
I20260129 14:55:59.898344 9475 tablet_bootstrap.cc:492] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3: Bootstrap starting.
I20260129 14:55:59.898577 9476 tablet_bootstrap.cc:654] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:59.898756 9477 tablet_bootstrap.cc:654] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:59.901049 9475 tablet_bootstrap.cc:654] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3: Neither blocks nor log segments found. Creating new log.
I20260129 14:55:59.903071 9476 tablet_bootstrap.cc:492] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67: No bootstrap required, opened a new log
I20260129 14:55:59.903225 9476 ts_tablet_manager.cc:1403] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260129 14:55:59.903916 9477 tablet_bootstrap.cc:492] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9: No bootstrap required, opened a new log
I20260129 14:55:59.904052 9477 ts_tablet_manager.cc:1403] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260129 14:55:59.904117 9476 raft_consensus.cc:359] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.904328 9476 raft_consensus.cc:385] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:59.904410 9476 raft_consensus.cc:740] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0721a8132fc84354ae4b9e42fe387c67, State: Initialized, Role: FOLLOWER
I20260129 14:55:59.904673 9476 consensus_queue.cc:260] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.904825 9477 raft_consensus.cc:359] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.905000 9477 raft_consensus.cc:385] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:59.905078 9477 raft_consensus.cc:740] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824b258cd009461bb4f25beea2af6eb9, State: Initialized, Role: FOLLOWER
I20260129 14:55:59.905741 9477 consensus_queue.cc:260] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.908013 9477 ts_tablet_manager.cc:1434] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260129 14:55:59.908233 9476 ts_tablet_manager.cc:1434] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260129 14:55:59.912091 9475 tablet_bootstrap.cc:492] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3: No bootstrap required, opened a new log
I20260129 14:55:59.912262 9475 ts_tablet_manager.cc:1403] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20260129 14:55:59.913282 9475 raft_consensus.cc:359] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.913471 9475 raft_consensus.cc:385] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:55:59.913558 9475 raft_consensus.cc:740] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f177394443439dbabb0738a08b8fc3, State: Initialized, Role: FOLLOWER
I20260129 14:55:59.913784 9475 consensus_queue.cc:260] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.914336 9475 ts_tablet_manager.cc:1434] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260129 14:55:59.946372 9436 raft_consensus.cc:493] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:55:59.946561 9436 raft_consensus.cc:515] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.947335 9436 leader_election.cc:290] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263), 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:59.947775 8599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "563c97415304403c8a50ff956fb54363" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0721a8132fc84354ae4b9e42fe387c67" is_pre_election: true
I20260129 14:55:59.947785 8462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "563c97415304403c8a50ff956fb54363" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9" is_pre_election: true
I20260129 14:55:59.947981 8599 raft_consensus.cc:2468] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 0.
I20260129 14:55:59.947995 8462 raft_consensus.cc:2468] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 0.
I20260129 14:55:59.948463 8669 leader_election.cc:304] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 824b258cd009461bb4f25beea2af6eb9, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:55:59.948705 9436 raft_consensus.cc:2804] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 14:55:59.948812 9436 raft_consensus.cc:493] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:55:59.948894 9436 raft_consensus.cc:3060] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:59.950580 9436 raft_consensus.cc:515] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
I20260129 14:55:59.951089 9436 leader_election.cc:290] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263), 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:55:59.951392 8599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "563c97415304403c8a50ff956fb54363" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0721a8132fc84354ae4b9e42fe387c67"
I20260129 14:55:59.951588 8599 raft_consensus.cc:3060] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:59.951617 8462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "563c97415304403c8a50ff956fb54363" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9"
I20260129 14:55:59.951817 8462 raft_consensus.cc:3060] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:55:59.954186 8599 raft_consensus.cc:2468] T 563c97415304403c8a50ff956fb54363 P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 1.
I20260129 14:55:59.954474 8462 raft_consensus.cc:2468] T 563c97415304403c8a50ff956fb54363 P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 1.
I20260129 14:55:59.954526 8668 leader_election.cc:304] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0721a8132fc84354ae4b9e42fe387c67, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:55:59.954819 9436 raft_consensus.cc:2804] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 14:55:59.955232 9436 raft_consensus.cc:697] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [term 1 LEADER]: Becoming Leader. State: Replica: e6f177394443439dbabb0738a08b8fc3, State: Running, Role: LEADER
I20260129 14:55:59.955565 9436 consensus_queue.cc:237] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } }
W20260129 14:56:00.054222 8534 connection.cc:537] server connection from 127.0.0.1:48964 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:00.054313 8532 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:00.054337 8670 connection.cc:537] server connection from 127.0.0.1:49544 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:00.054562 8667 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:00.054574 8395 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:00.055809 8396 connection.cc:537] server connection from 127.0.0.1:46128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:00.056174 8786 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:00.056756 8786 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.253:34637)
W20260129 14:56:00.057092 8786 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.171.253:34637)
W20260129 14:56:00.057400 9440 negotiation.cc:337] Failed RPC negotiation. Trace:
0129 14:56:00.055185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.171.253:34637 (local address 127.7.171.195:46543)
0129 14:56:00.055440 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0129 14:56:00.055460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0129 14:56:00.055579 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0129 14:56:00.056462 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130}
W20260129 14:56:00.057885 8786 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.7.171.253:34637)
W20260129 14:56:00.057925 9441 negotiation.cc:337] Failed RPC negotiation. Trace:
0129 14:56:00.055377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.171.253:34637 (local address 127.7.171.193:51041)
0129 14:56:00.055621 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0129 14:56:00.055639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0129 14:56:00.055765 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0129 14:56:00.057146 (+ 1381us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":128}
W20260129 14:56:00.058985 9443 negotiation.cc:337] Failed RPC negotiation. Trace:
0129 14:56:00.055361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.171.253:34637 (local address 127.7.171.194:40619)
0129 14:56:00.055578 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0129 14:56:00.055593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0129 14:56:00.055694 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0129 14:56:00.058419 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127}
I20260129 14:56:00.363410 9489 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c63d8c4ffa4047c9a8423771f4723d6e)
I20260129 14:56:00.363631 9489 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:00.364312 9489 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:00.366026 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:00.366281 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:00.366437 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:00.366904 9489 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 14:56:00.754035 9494 consensus_queue.cc:1048] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:56:00.759738 9498 consensus_queue.cc:1048] T 563c97415304403c8a50ff956fb54363 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:56:00.890720 9505 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c63d8c4ffa4047c9a8423771f4723d6e)
I20260129 14:56:00.890900 9505 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:00.891592 9505 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:00.893857 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
W20260129 14:56:00.894006 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:00.894200 9077 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:00.894448 9505 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 14:56:01.565752 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.253:34637
--webserver_interface=127.7.171.253
--webserver_port=41757
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260129 14:56:01.626521 9519 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c63d8c4ffa4047c9a8423771f4723d6e)
I20260129 14:56:01.626706 9519 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:01.627324 9519 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:01.628983 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:01.629196 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:01.629287 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:01.629503 9519 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 14:56:01.730578 9059 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111),
I20260129 14:56:01.730794 9059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260129 14:56:01.877849 9515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:56:01.878162 9515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:56:01.878237 9515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:56:01.886422 9515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:56:01.886513 9515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:56:01.886567 9515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:56:01.886615 9515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:56:01.896291 9515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.253:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.171.253
--webserver_port=41757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:56:01.897840 9515 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:56:01.899466 9515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:56:01.906877 9533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:01.907006 9532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:01.907491 9535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:56:01.908182 9515 server_base.cc:1060] running on GCE node
I20260129 14:56:01.908824 9515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:56:01.909981 9515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:56:01.911180 9515 hybrid_clock.cc:648] HybridClock initialized: now 1769698561911138 us; error 33 us; skew 500 ppm
I20260129 14:56:01.911489 9515 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:56:01.914251 9515 webserver.cc:492] Webserver started at http://127.7.171.253:41757/ using document root <none> and password file <none>
I20260129 14:56:01.914969 9515 fs_manager.cc:362] Metadata directory not provided
I20260129 14:56:01.915091 9515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:56:01.919912 9515 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260129 14:56:01.922600 9541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:56:01.923775 9515 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 14:56:01.923933 9515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
uuid: "c63d8c4ffa4047c9a8423771f4723d6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:56:01.924592 9515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:56:01.945878 9515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:56:01.946566 9515 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:56:01.946800 9515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:56:01.964957 9515 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.253:34637
I20260129 14:56:01.964994 9592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.253:34637 every 8 connection(s)
I20260129 14:56:01.966363 9515 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-1/data/info.pb
I20260129 14:56:01.972093 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 9515
I20260129 14:56:01.972594 9593 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:56:01.977082 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Bootstrap starting.
I20260129 14:56:01.987911 9593 log.cc:826] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:56:01.999504 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=8 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 14:56:01.999939 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Bootstrap complete.
I20260129 14:56:02.004467 9593 raft_consensus.cc:359] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:02.005092 9593 raft_consensus.cc:740] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63d8c4ffa4047c9a8423771f4723d6e, State: Initialized, Role: FOLLOWER
I20260129 14:56:02.005643 9593 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 5.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:02.006327 9598 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:02.006587 9598 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:02.007197 9593 sys_catalog.cc:565] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:56:02.028410 9612 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:02.028606 9612 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: loading cluster ID for follower catalog manager: success
I20260129 14:56:02.032757 9612 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: acquiring CA information for follower catalog manager: success
I20260129 14:56:02.034902 9612 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:56:02.038717 9515 master_runner.cc:186] Error getting master registration for 127.7.171.252:43411: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111),
I20260129 14:56:02.042173 9515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:02.089139 8781 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:56:02.091768 9557 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:59285; Asking this server to re-register.
I20260129 14:56:02.092403 8781 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:02.093654 9557 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:56:02.272625 8646 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:56:02.274065 8512 heartbeater.cc:344] Connected to a master server at 127.7.171.253:34637
I20260129 14:56:02.274031 9557 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:48575; Asking this server to re-register.
I20260129 14:56:02.274550 8646 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:02.275344 9557 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:50021; Asking this server to re-register.
I20260129 14:56:02.275831 8512 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:02.275841 9558 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:56:02.276830 9558 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:56:02.394963 9598 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:02.395236 9598 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:02.397084 9598 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:56:02.397488 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 6 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e" is_pre_election: true
I20260129 14:56:02.397749 9105 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63d8c4ffa4047c9a8423771f4723d6e in term 5.
W20260129 14:56:02.397723 9543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:02.398522 9543 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e; no voters:
I20260129 14:56:02.398842 9598 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 14:56:02.398973 9598 raft_consensus.cc:493] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:56:02.399081 9598 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 5 FOLLOWER]: Advancing to term 6
W20260129 14:56:02.399324 9543 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:02.402794 9598 raft_consensus.cc:515] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:02.403316 9598 leader_election.cc:290] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 6 election: Requested vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:56:02.403671 9105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" candidate_term: 6 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: false dest_uuid: "5cde2b69b78d4e8491572266f7019c6e"
I20260129 14:56:02.403891 9105 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 5 FOLLOWER]: Advancing to term 6
W20260129 14:56:02.404439 9543 leader_election.cc:336] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:02.405735 9105 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63d8c4ffa4047c9a8423771f4723d6e in term 6.
I20260129 14:56:02.406167 9543 leader_election.cc:304] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2
I20260129 14:56:02.406407 9598 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 6 FOLLOWER]: Leader election won for term 6
I20260129 14:56:02.406774 9598 raft_consensus.cc:697] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [term 6 LEADER]: Becoming Leader. State: Replica: c63d8c4ffa4047c9a8423771f4723d6e, State: Running, Role: LEADER
I20260129 14:56:02.407202 9598 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 5.13, Last appended by leader: 13, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:02.409683 9624 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63d8c4ffa4047c9a8423771f4723d6e. Latest consensus state: current_term: 6 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:02.409902 9624 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:56:02.410497 9626 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:56:02.412802 9626 catalog_manager.cc:679] Loaded metadata for table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]
I20260129 14:56:02.413220 9626 catalog_manager.cc:679] Loaded metadata for table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]
I20260129 14:56:02.414637 9626 tablet_loader.cc:96] loaded metadata for tablet 563c97415304403c8a50ff956fb54363 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed])
I20260129 14:56:02.415136 9626 tablet_loader.cc:96] loaded metadata for tablet d7c50ee4a0464ec4a03a6a30e4cb74df (table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc])
I20260129 14:56:02.415366 9626 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:56:02.416160 9626 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:02.416225 9626 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:56:02.417251 9626 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:56:02.418020 9626 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Loaded TSK: 0
I20260129 14:56:02.418795 9626 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:56:02.419515 9626 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:56:02.420303 9626 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:56:02.454284 9558 master_service.cc:954] ReplaceTablet: received request to replace tablet d7c50ee4a0464ec4a03a6a30e4cb74df from {username='slave'} at 127.0.0.1:60544
I20260129 14:56:02.454351 9557 master_service.cc:954] ReplaceTablet: received request to replace tablet d7c50ee4a0464ec4a03a6a30e4cb74df from {username='slave'} at 127.0.0.1:60544
I20260129 14:56:02.454901 9556 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_ffffe38dc46a4a97ba5f4b081cbcc822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.455066 9553 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.455942 9554 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.456419 9555 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.456465 9551 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.457052 9550 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.457473 9552 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_b6e29b73e03042138b5a216f08dce0c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.457748 9549 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.460551 9105 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Refusing update from remote peer c63d8c4ffa4047c9a8423771f4723d6e: Log matching property violated. Preceding OpId in replica: term: 5 index: 13. Preceding OpId from leader: term: 6 index: 15. (index mismatch)
I20260129 14:56:02.461356 9624 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260129 14:56:02.461879 9543 consensus_peers.cc:597] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e -> Peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Couldn't send request to peer 8db7ebac21af437fbdf8e1bd7e00fca2. Status: Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20260129 14:56:02.462337 9611 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_c2b80faabf21435d991d592f638440aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2b80faabf21435d991d592f638440aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2b80faabf21435d991d592f638440aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_c2b80faabf21435d991d592f638440aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2b80faabf21435d991d592f638440aa\",\"kudu.table_id\":\"908853f113ee48df891ae3a9db09c1ed\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_95ece788510d4811b741bfafc484dd89, message={"server":"","servicePrincipal":"","db":"default","table":"table_95ece788510d4811b741bfafc484dd89","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_95ece788510d4811b741bfafc484dd89\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_95ece788510d4811b741bfafc484dd89\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_95ece788510d4811b741bfafc484dd89\",\"kudu.table_id\":\"66667881263c4bb0a181c0c7576354f7\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f79e5de1dcd google::LogMessage::Fail() at ??:0
I20260129 14:56:02.464847 9628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63d8c4ffa4047c9a8423771f4723d6e. Latest consensus state: current_term: 6 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:02.465075 9628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f79e5de5b93 google::LogMessage::SendToLog() at ??:0
I20260129 14:56:02.466516 9627 mvcc.cc:204] Tried to move back new op lower bound from 7248685311831785472 to 7248685311623266304. Current Snapshot: MvccSnapshot[applied={T|T < 7248685311831785472}]
@ 0x7f79e5de17cc google::LogMessage::Flush() at ??:0
@ 0x7f79e5de1a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f79fc7d87f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260129 14:56:02.480497 9628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:02.480688 9628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:02.480587 9598 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:02.480844 9598 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:56:02.482276 9558 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c63d8c4ffa4047c9a8423771f4723d6e: Sending DeleteTablet for 3 replicas of tablet d7c50ee4a0464ec4a03a6a30e4cb74df
I20260129 14:56:02.482997 9557 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_60640f05a979404380083e385f1e6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:02.484401 9558 catalog_manager.cc:6724] ReplaceTablet: tablet d7c50ee4a0464ec4a03a6a30e4cb74df deleted and replaced by tablet 740d94adef8b493bbbb88380b11c3404
I20260129 14:56:02.484835 9558 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "default.Table_17c52877671f4e30bcf4f682122e181e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f79fc7d70a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f79fc7ded1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f79fc7deb99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
I20260129 14:56:02.516552 8714 tablet_service.cc:1552] Processing DeleteTablet for tablet d7c50ee4a0464ec4a03a6a30e4cb74df with delete_type TABLET_DATA_DELETED (replaced by tablet 740d94adef8b493bbbb88380b11c3404) from {username='slave'} at 127.0.0.1:49552
I20260129 14:56:02.516741 8578 tablet_service.cc:1552] Processing DeleteTablet for tablet d7c50ee4a0464ec4a03a6a30e4cb74df with delete_type TABLET_DATA_DELETED (replaced by tablet 740d94adef8b493bbbb88380b11c3404) from {username='slave'} at 127.0.0.1:48966
I20260129 14:56:02.517110 8579 tablet_service.cc:1505] Processing CreateTablet for tablet 740d94adef8b493bbbb88380b11c3404 (DEFAULT_TABLE table=default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:56:02.517701 8579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740d94adef8b493bbbb88380b11c3404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:56:02.522616 8713 tablet_service.cc:1505] Processing CreateTablet for tablet 740d94adef8b493bbbb88380b11c3404 (DEFAULT_TABLE table=default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:56:02.523103 8713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740d94adef8b493bbbb88380b11c3404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 14:56:02.529245 9648 tablet_bootstrap.cc:492] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67: Bootstrap starting.
I20260129 14:56:02.529744 8441 tablet_service.cc:1552] Processing DeleteTablet for tablet d7c50ee4a0464ec4a03a6a30e4cb74df with delete_type TABLET_DATA_DELETED (replaced by tablet 740d94adef8b493bbbb88380b11c3404) from {username='slave'} at 127.0.0.1:46136
I20260129 14:56:02.530252 9647 tablet_replica.cc:333] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: stopping tablet replica
I20260129 14:56:02.530499 9647 raft_consensus.cc:2243] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 1 LEADER]: Raft consensus shutting down.
I20260129 14:56:02.530748 8442 tablet_service.cc:1505] Processing CreateTablet for tablet 740d94adef8b493bbbb88380b11c3404 (DEFAULT_TABLE table=default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 14:56:02.531163 9647 raft_consensus.cc:2272] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 14:56:02.531251 8442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740d94adef8b493bbbb88380b11c3404. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f79e8b56f99 kudu::Thread::SuperviseThread() at ??:0
I20260129 14:56:02.531893 9647 ts_tablet_manager.cc:1916] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f79ec1636db start_thread at ??:0
@ 0x7f79e4c9271f clone at ??:0
I20260129 14:56:02.536881 9649 tablet_replica.cc:333] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: stopping tablet replica
I20260129 14:56:02.537092 9649 raft_consensus.cc:2243] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 14:56:02.537312 9649 raft_consensus.cc:2272] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 14:56:02.537806 9649 ts_tablet_manager.cc:1916] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 14:56:02.538230 9648 tablet_bootstrap.cc:654] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67: Neither blocks nor log segments found. Creating new log.
I20260129 14:56:02.539613 9650 tablet_bootstrap.cc:492] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3: Bootstrap starting.
I20260129 14:56:02.540544 9646 tablet_replica.cc:333] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: stopping tablet replica
I20260129 14:56:02.540795 9646 raft_consensus.cc:2243] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 14:56:02.541043 9646 raft_consensus.cc:2272] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 14:56:02.541265 9650 tablet_bootstrap.cc:654] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3: Neither blocks nor log segments found. Creating new log.
I20260129 14:56:02.541656 9646 ts_tablet_manager.cc:1916] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 14:56:02.543722 9647 ts_tablet_manager.cc:1929] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 14:56:02.543802 9647 log.cc:1199] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Deleting WAL directory at /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-1/wal/wals/d7c50ee4a0464ec4a03a6a30e4cb74df
I20260129 14:56:02.544143 9647 ts_tablet_manager.cc:1950] T d7c50ee4a0464ec4a03a6a30e4cb74df P 0721a8132fc84354ae4b9e42fe387c67: Deleting consensus metadata
I20260129 14:56:02.545751 9650 tablet_bootstrap.cc:492] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3: No bootstrap required, opened a new log
I20260129 14:56:02.545871 9650 ts_tablet_manager.cc:1403] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260129 14:56:02.546624 9650 raft_consensus.cc:359] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.546800 9650 raft_consensus.cc:385] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:56:02.546891 9650 raft_consensus.cc:740] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f177394443439dbabb0738a08b8fc3, State: Initialized, Role: FOLLOWER
I20260129 14:56:02.547642 9648 tablet_bootstrap.cc:492] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67: No bootstrap required, opened a new log
I20260129 14:56:02.547771 9648 ts_tablet_manager.cc:1403] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260129 14:56:02.547781 9646 ts_tablet_manager.cc:1929] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 14:56:02.547886 9646 log.cc:1199] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: Deleting WAL directory at /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-2/wal/wals/d7c50ee4a0464ec4a03a6a30e4cb74df
I20260129 14:56:02.548295 9646 ts_tablet_manager.cc:1950] T d7c50ee4a0464ec4a03a6a30e4cb74df P e6f177394443439dbabb0738a08b8fc3: Deleting consensus metadata
I20260129 14:56:02.548542 9648 raft_consensus.cc:359] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.548691 9648 raft_consensus.cc:385] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:56:02.548755 9648 raft_consensus.cc:740] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0721a8132fc84354ae4b9e42fe387c67, State: Initialized, Role: FOLLOWER
I20260129 14:56:02.549089 9648 consensus_queue.cc:260] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.549268 9651 tablet_bootstrap.cc:492] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9: Bootstrap starting.
I20260129 14:56:02.549508 9650 consensus_queue.cc:260] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.550750 9648 ts_tablet_manager.cc:1434] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260129 14:56:02.551286 9651 tablet_bootstrap.cc:654] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9: Neither blocks nor log segments found. Creating new log.
I20260129 14:56:02.554525 9650 ts_tablet_manager.cc:1434] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20260129 14:56:02.555433 9649 ts_tablet_manager.cc:1929] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 14:56:02.555575 9649 log.cc:1199] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Deleting WAL directory at /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/ts-0/wal/wals/d7c50ee4a0464ec4a03a6a30e4cb74df
I20260129 14:56:02.556093 9649 ts_tablet_manager.cc:1950] T d7c50ee4a0464ec4a03a6a30e4cb74df P 824b258cd009461bb4f25beea2af6eb9: Deleting consensus metadata
I20260129 14:56:02.558012 9651 tablet_bootstrap.cc:492] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9: No bootstrap required, opened a new log
I20260129 14:56:02.558140 9651 ts_tablet_manager.cc:1403] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260129 14:56:02.558948 9651 raft_consensus.cc:359] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.559128 9651 raft_consensus.cc:385] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 14:56:02.559226 9651 raft_consensus.cc:740] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824b258cd009461bb4f25beea2af6eb9, State: Initialized, Role: FOLLOWER
I20260129 14:56:02.559438 9651 consensus_queue.cc:260] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.560263 9651 ts_tablet_manager.cc:1434] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260129 14:56:02.697062 8668 connection.cc:537] server connection from 127.0.0.1:49552 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:02.697084 8535 connection.cc:537] server connection from 127.0.0.1:48966 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:02.697110 8398 connection.cc:537] server connection from 127.0.0.1:46136 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:02.697322 8532 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:02.697391 8786 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:02.697777 8395 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:02.698378 8786 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.253:34637)
W20260129 14:56:02.698737 8786 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.171.253:34637)
W20260129 14:56:02.699172 8667 connection.cc:537] client connection to 127.7.171.253:34637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:02.699747 8786 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.7.171.253:34637)
I20260129 14:56:02.754812 9655 raft_consensus.cc:493] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:02.754968 9655 raft_consensus.cc:515] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.755669 9655 leader_election.cc:290] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151), 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:56:02.756150 8462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740d94adef8b493bbbb88380b11c3404" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9" is_pre_election: true
I20260129 14:56:02.756196 8599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740d94adef8b493bbbb88380b11c3404" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0721a8132fc84354ae4b9e42fe387c67" is_pre_election: true
I20260129 14:56:02.756340 8462 raft_consensus.cc:2468] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 0.
I20260129 14:56:02.756390 8599 raft_consensus.cc:2468] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 0.
I20260129 14:56:02.756775 8669 leader_election.cc:304] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 824b258cd009461bb4f25beea2af6eb9, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:56:02.757028 9655 raft_consensus.cc:2804] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 14:56:02.757135 9655 raft_consensus.cc:493] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:56:02.757205 9655 raft_consensus.cc:3060] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:56:02.758934 9655 raft_consensus.cc:515] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:02.759402 9655 leader_election.cc:290] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151), 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:56:02.759807 8462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740d94adef8b493bbbb88380b11c3404" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824b258cd009461bb4f25beea2af6eb9"
I20260129 14:56:02.759860 8599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740d94adef8b493bbbb88380b11c3404" candidate_uuid: "e6f177394443439dbabb0738a08b8fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0721a8132fc84354ae4b9e42fe387c67"
I20260129 14:56:02.759967 8462 raft_consensus.cc:3060] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:56:02.760022 8599 raft_consensus.cc:3060] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67 [term 0 FOLLOWER]: Advancing to term 1
I20260129 14:56:02.761758 8462 raft_consensus.cc:2468] T 740d94adef8b493bbbb88380b11c3404 P 824b258cd009461bb4f25beea2af6eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 1.
I20260129 14:56:02.761812 8599 raft_consensus.cc:2468] T 740d94adef8b493bbbb88380b11c3404 P 0721a8132fc84354ae4b9e42fe387c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f177394443439dbabb0738a08b8fc3 in term 1.
I20260129 14:56:02.762120 8669 leader_election.cc:304] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 824b258cd009461bb4f25beea2af6eb9, e6f177394443439dbabb0738a08b8fc3; no voters:
I20260129 14:56:02.762424 9655 raft_consensus.cc:2804] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 14:56:02.762801 9655 raft_consensus.cc:697] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [term 1 LEADER]: Becoming Leader. State: Replica: e6f177394443439dbabb0738a08b8fc3, State: Running, Role: LEADER
I20260129 14:56:02.763159 9655 consensus_queue.cc:237] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 } } peers { permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" member_type: VOTER last_known_addr { host: "127.7.171.195" port: 44395 } } peers { permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 } }
I20260129 14:56:03.011915 9667 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c63d8c4ffa4047c9a8423771f4723d6e)
I20260129 14:56:03.012104 9667 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:03.012784 9667 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:03.014537 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:03.014688 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:03.014824 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:03.015051 9667 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 14:56:03.610606 9671 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c63d8c4ffa4047c9a8423771f4723d6e)
I20260129 14:56:03.610787 9671 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:03.611423 9671 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:03.613274 9077 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:03.613433 9078 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:03.613562 9078 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:03.613794 9671 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 14:56:03.706171 9675 consensus_queue.cc:1048] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" member_type: VOTER last_known_addr { host: "127.7.171.193" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 14:56:03.712846 9681 consensus_queue.cc:1048] T 740d94adef8b493bbbb88380b11c3404 P e6f177394443439dbabb0738a08b8fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" member_type: VOTER last_known_addr { host: "127.7.171.194" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 14:56:04.094352 7855 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu pid 9059
I20260129 14:56:04.103678 9125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260129 14:56:04.209518 7855 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu with pid 9059
W20260129 14:56:04.229907 8533 connection.cc:537] client connection to 127.7.171.254:38695 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:04.229899 8668 connection.cc:537] client connection to 127.7.171.254:38695 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:04.230103 8396 connection.cc:537] client connection to 127.7.171.254:38695 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260129 14:56:04.962507 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--webserver_interface=127.7.171.254
--webserver_port=33955
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:56:05.283470 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:56:05.283771 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:56:05.283907 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:56:05.291824 9703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:56:05.291904 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:56:05.291972 9703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:56:05.292042 9703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:56:05.301775 9703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.171.254
--webserver_port=33955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:56:05.303260 9703 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:56:05.305022 9703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:56:05.313963 9711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:05.313963 9710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:05.314102 9713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:56:05.314781 9703 server_base.cc:1060] running on GCE node
I20260129 14:56:05.315428 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:56:05.316694 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:56:05.317904 9703 hybrid_clock.cc:648] HybridClock initialized: now 1769698565317844 us; error 43 us; skew 500 ppm
I20260129 14:56:05.318236 9703 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:56:05.320194 9703 webserver.cc:492] Webserver started at http://127.7.171.254:33955/ using document root <none> and password file <none>
I20260129 14:56:05.320868 9703 fs_manager.cc:362] Metadata directory not provided
I20260129 14:56:05.320969 9703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:56:05.326821 9703 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260129 14:56:05.329452 9719 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:56:05.330729 9703 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260129 14:56:05.330883 9703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
uuid: "5cde2b69b78d4e8491572266f7019c6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:56:05.331514 9703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:56:05.357240 9703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:56:05.358045 9703 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:56:05.358309 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:56:05.377029 9703 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.254:38695
I20260129 14:56:05.377074 9770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.254:38695 every 8 connection(s)
I20260129 14:56:05.378410 9703 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
I20260129 14:56:05.380801 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 9703
I20260129 14:56:05.385421 9771 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:56:05.388464 9771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap starting.
I20260129 14:56:05.397235 9771 log.cc:826] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:56:05.409936 9771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=9 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 14:56:05.410442 9771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap complete.
I20260129 14:56:05.416316 9771 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:05.416991 9771 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Initialized, Role: FOLLOWER
I20260129 14:56:05.417574 9771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:05.418370 9778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:05.418781 9778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:05.419602 9771 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:56:05.421187 8780 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:56:05.438310 9792 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:05.438522 9792 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: loading cluster ID for follower catalog manager: success
I20260129 14:56:05.442946 9792 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: acquiring CA information for follower catalog manager: success
I20260129 14:56:05.447016 9792 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:56:05.454799 9703 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111),
I20260129 14:56:05.458580 9703 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:05.740324 8645 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:56:05.740943 8508 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:56:05.742307 9735 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:57745; Asking this server to re-register.
I20260129 14:56:05.742385 9736 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:35841; Asking this server to re-register.
I20260129 14:56:05.742903 8645 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:05.743007 8508 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:05.744582 9736 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:56:05.744683 9735 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:56:05.923142 9800 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:05.923370 9800 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:05.925284 9800 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:56:05.925606 9736 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:45147; Asking this server to re-register.
W20260129 14:56:05.925846 9720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:05.926213 8780 heartbeater.cc:461] Registering TS with master...
W20260129 14:56:05.927332 9721 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:05.927438 9736 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
W20260129 14:56:05.927520 9720 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:05.927695 9720 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:05.928104 9800 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 14:56:06.652768 9806 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:06.652940 9806 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:06.653620 9806 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:06.655453 9720 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:06.655625 9721 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:06.655825 9721 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:06.656042 9806 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 14:56:07.632671 9823 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:07.632866 9823 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:07.633576 9823 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:07.635318 9720 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:07.635478 9721 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411): Network error: Client connection negotiation failed: client connection to 127.7.171.252:43411: connect: Connection refused (error 111)
I20260129 14:56:07.635588 9721 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e; no voters: 8db7ebac21af437fbdf8e1bd7e00fca2, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:07.635768 9823 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 14:56:08.585131 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--webserver_interface=127.7.171.252
--webserver_port=37043
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:56:08.886533 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:56:08.886811 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:56:08.886884 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:56:08.894836 9839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:56:08.894913 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:56:08.894965 9839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:56:08.895012 9839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:56:08.904028 9839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.171.252
--webserver_port=37043
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:56:08.905537 9839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:56:08.907078 9839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:56:08.914557 9849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:08.914516 9850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:08.915594 9852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:56:08.916011 9839 server_base.cc:1060] running on GCE node
I20260129 14:56:08.916755 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:56:08.917963 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:56:08.919160 9839 hybrid_clock.cc:648] HybridClock initialized: now 1769698568919128 us; error 28 us; skew 500 ppm
I20260129 14:56:08.919443 9839 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:56:08.921526 9839 webserver.cc:492] Webserver started at http://127.7.171.252:37043/ using document root <none> and password file <none>
I20260129 14:56:08.922132 9839 fs_manager.cc:362] Metadata directory not provided
I20260129 14:56:08.922230 9839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:56:08.927706 9839 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260129 14:56:08.930742 9860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:56:08.932206 9839 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260129 14:56:08.932358 9839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
format_stamp: "Formatted at 2026-01-29 14:55:44 on dist-test-slave-n0m6"
I20260129 14:56:08.932931 9839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:56:08.950374 9839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:56:08.951118 9839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:56:08.951385 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:56:08.969753 9839 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.252:43411
I20260129 14:56:08.969784 9911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.252:43411 every 8 connection(s)
I20260129 14:56:08.971230 9839 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
I20260129 14:56:08.977473 9912 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:56:08.980190 9912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap starting.
I20260129 14:56:08.981460 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 9839
I20260129 14:56:08.988675 9912 log.cc:826] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Log is configured to *not* fsync() on all Append() calls
I20260129 14:56:08.999625 9912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=0} inserts{seen=7 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260129 14:56:09.000120 9912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap complete.
I20260129 14:56:09.004534 9912 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:09.005244 9912 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db7ebac21af437fbdf8e1bd7e00fca2, State: Initialized, Role: FOLLOWER
I20260129 14:56:09.005708 9912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 4.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:09.006358 9916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:09.006573 9916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:09.007119 9912 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:56:09.030671 9930 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:09.030787 9930 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: loading cluster ID for follower catalog manager: success
I20260129 14:56:09.034854 9930 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: acquiring CA information for follower catalog manager: success
I20260129 14:56:09.037197 9930 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:56:09.037328 9839 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111),
I20260129 14:56:09.040731 9839 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:09.210964 9935 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:09.211144 9935 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:09.211839 9935 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:09.214702 9720 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:09.220465 9887 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" is_pre_election: true
I20260129 14:56:09.221151 9887 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 4.
I20260129 14:56:09.221792 9721 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:09.222240 9935 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 14:56:09.222369 9935 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:56:09.222481 9935 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 6 FOLLOWER]: Advancing to term 7
I20260129 14:56:09.227416 9935 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:09.228547 9935 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 election: Requested vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:56:09.228796 9887 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
I20260129 14:56:09.229014 9887 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 4 FOLLOWER]: Advancing to term 7
I20260129 14:56:09.239432 9887 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 7.
I20260129 14:56:09.240731 9721 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters:
I20260129 14:56:09.241077 9935 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Leader election won for term 7
I20260129 14:56:09.245646 9935 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 LEADER]: Becoming Leader. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Running, Role: LEADER
I20260129 14:56:09.246165 9935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
W20260129 14:56:09.253688 9720 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:09.268179 9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde2b69b78d4e8491572266f7019c6e. Latest consensus state: current_term: 7 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:09.268508 9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:56:09.273132 8507 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:56:09.273561 9946 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:56:09.274006 8644 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:56:09.276554 9877 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:57495; Asking this server to re-register.
I20260129 14:56:09.276834 9946 catalog_manager.cc:679] Loaded metadata for table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]
I20260129 14:56:09.277377 9946 catalog_manager.cc:679] Loaded metadata for table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]
I20260129 14:56:09.277522 8507 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:09.277719 9877 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:58387; Asking this server to re-register.
I20260129 14:56:09.278548 8644 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:09.279440 9946 tablet_loader.cc:96] loaded metadata for tablet 563c97415304403c8a50ff956fb54363 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed])
I20260129 14:56:09.279800 9946 tablet_loader.cc:96] loaded metadata for tablet 740d94adef8b493bbbb88380b11c3404 (table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc])
I20260129 14:56:09.279340 9877 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:56:09.280179 9876 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:56:09.280644 9946 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:56:09.281598 9946 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:09.281667 9946 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:56:09.282711 9946 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:56:09.283483 9946 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Loaded TSK: 0
I20260129 14:56:09.284296 9946 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:56:09.285030 9946 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:56:09.285791 9946 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:56:09.299510 9735 master_service.cc:954] ReplaceTablet: received request to replace tablet 740d94adef8b493bbbb88380b11c3404 from {username='slave'} at 127.0.0.1:33752
I20260129 14:56:09.300621 9733 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.301769 9734 master_service.cc:954] ReplaceTablet: received request to replace tablet 7b5bc5c45b2a4314b92c797aac0ca371 from {username='slave'} at 127.0.0.1:33752
I20260129 14:56:09.302482 9731 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.302646 9734 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.301775 9727 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.303234 9732 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.304219 9729 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_b6e29b73e03042138b5a216f08dce0c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.304271 9728 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.304816 9730 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "default.Table_ffffe38dc46a4a97ba5f4b081cbcc822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:09.305460 9736 master_service.cc:954] ReplaceTablet: received request to replace tablet 563c97415304403c8a50ff956fb54363 from {username='slave'} at 127.0.0.1:33752
F20260129 14:56:09.307685 9791 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_c2b80faabf21435d991d592f638440aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2b80faabf21435d991d592f638440aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2b80faabf21435d991d592f638440aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_c2b80faabf21435d991d592f638440aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2b80faabf21435d991d592f638440aa\",\"kudu.table_id\":\"908853f113ee48df891ae3a9db09c1ed\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_95ece788510d4811b741bfafc484dd89, message={"server":"","servicePrincipal":"","db":"default","table":"table_95ece788510d4811b741bfafc484dd89","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_95ece788510d4811b741bfafc484dd89\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_95ece788510d4811b741bfafc484dd89\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_95ece788510d4811b741bfafc484dd89\",\"kudu.table_id\":\"66667881263c4bb0a181c0c7576354f7\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fa132313dcd google::LogMessage::Fail() at ??:0
@ 0x7fa132317b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa1323137cc google::LogMessage::Flush() at ??:0
I20260129 14:56:09.315517 9887 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Refusing update from remote peer 5cde2b69b78d4e8491572266f7019c6e: Log matching property violated. Preceding OpId in replica: term: 4 index: 10. Preceding OpId from leader: term: 7 index: 18. (index mismatch)
@ 0x7fa132313a11 google::LogMessage::~LogMessage() at ??:0
I20260129 14:56:09.316641 9935 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 9, Time since last communication: 0.000s
I20260129 14:56:09.316972 9935 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Peer 8db7ebac21af437fbdf8e1bd7e00fca2 log is divergent from this leader: its last log entry 4.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260129 14:56:09.320086 9720 consensus_peers.cc:597] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e -> Peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Couldn't send request to peer c63d8c4ffa4047c9a8423771f4723d6e. Status: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 14:56:09.321048 9887 pending_rounds.cc:85] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Aborting all ops after (but not including) 9
I20260129 14:56:09.321184 9887 pending_rounds.cc:107] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Aborting uncommitted NO_OP operation due to leader change: 4.10
I20260129 14:56:09.321341 9887 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7fa148d0a7f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fa148d090a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fa148d10d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fa148d10b99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fa135088f99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa1386956db start_thread at ??:0
@ 0x7fa1311c471f clone at ??:0
I20260129 14:56:09.363023 9916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde2b69b78d4e8491572266f7019c6e. Latest consensus state: current_term: 7 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:09.363327 9916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:09.363951 9955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:09.364302 9955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:09.364622 9949 mvcc.cc:204] Tried to move back new op lower bound from 7248685311978975232 to 7248685253213122560. Current Snapshot: MvccSnapshot[applied={T|T < 7248685311978975232 or (T in {7248685311978975232})}]
I20260129 14:56:09.364784 9957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:09.364734 9953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:09.365209 9957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:09.365296 9953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:09.437482 8779 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:56:09.438920 9876 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:55319; Asking this server to re-register.
I20260129 14:56:09.439404 8779 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:09.440471 9876 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
W20260129 14:56:09.559924 9862 connection.cc:537] server connection from 127.0.0.1:36158 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:09.560194 8787 connection.cc:537] client connection to 127.7.171.254:38695 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:09.560639 8668 connection.cc:537] client connection to 127.7.171.254:38695 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:09.560958 8787 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.254:38695)
W20260129 14:56:09.561302 8787 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.171.254:38695)
W20260129 14:56:09.562220 8787 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.7.171.254:38695)
I20260129 14:56:09.822844 9953 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:09.823037 9953 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:09.824615 9953 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:09.825341 9862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:09.826386 9861 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:09.826550 9862 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:09.826653 9862 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:09.826912 9953 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 14:56:10.379626 9972 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:10.379820 9972 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:10.380537 9972 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:10.382463 9862 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:10.382607 9861 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:10.382753 9861 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:10.382988 9972 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 14:56:11.114818 9978 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:11.114992 9978 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:11.115592 9978 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
I20260129 14:56:11.116734 7855 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu pid 9839
W20260129 14:56:11.117753 9861 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:11.118019 9862 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:11.118165 9862 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:11.118459 9978 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 14:56:11.123696 9907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260129 14:56:11.225677 7855 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu with pid 9839
I20260129 14:56:11.836205 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--webserver_interface=127.7.171.252
--webserver_port=37043
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:56:12.138971 9990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:56:12.139261 9990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:56:12.139364 9990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:56:12.147171 9990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:56:12.147277 9990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:56:12.147367 9990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:56:12.147426 9990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:56:12.156596 9990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.252:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.171.252
--webserver_port=37043
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:56:12.158054 9990 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:56:12.159672 9990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:56:12.167404 9995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:12.167433 9996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:12.167956 9998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:56:12.168368 9990 server_base.cc:1060] running on GCE node
I20260129 14:56:12.168999 9990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:56:12.170182 9990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:56:12.171376 9990 hybrid_clock.cc:648] HybridClock initialized: now 1769698572171333 us; error 39 us; skew 500 ppm
I20260129 14:56:12.171693 9990 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:56:12.173686 9990 webserver.cc:492] Webserver started at http://127.7.171.252:37043/ using document root <none> and password file <none>
I20260129 14:56:12.174327 9990 fs_manager.cc:362] Metadata directory not provided
I20260129 14:56:12.174410 9990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:56:12.179402 9990 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260129 14:56:12.182258 10004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:56:12.183476 9990 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260129 14:56:12.183626 9990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
format_stamp: "Formatted at 2026-01-29 14:55:44 on dist-test-slave-n0m6"
I20260129 14:56:12.184348 9990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:56:12.201510 9990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:56:12.202232 9990 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:56:12.202469 9990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:56:12.221113 9990 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.252:43411
I20260129 14:56:12.221134 10055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.252:43411 every 8 connection(s)
I20260129 14:56:12.222676 9990 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-2/data/info.pb
I20260129 14:56:12.230234 10056 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:56:12.232604 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 9990
I20260129 14:56:12.233347 10056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap starting.
I20260129 14:56:12.243666 10056 log.cc:826] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Log is configured to *not* fsync() on all Append() calls
I20260129 14:56:12.261646 10056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=1 applied=16 ignored=0} inserts{seen=9 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260129 14:56:12.262264 10056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: Bootstrap complete.
I20260129 14:56:12.267530 10056 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:12.270324 10056 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db7ebac21af437fbdf8e1bd7e00fca2, State: Initialized, Role: FOLLOWER
I20260129 14:56:12.270906 10056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 7.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:12.271600 10061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:12.271801 10061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:12.272657 10056 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:56:12.285388 10076 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:12.285583 10076 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: loading cluster ID for follower catalog manager: success
I20260129 14:56:12.290232 10076 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: acquiring CA information for follower catalog manager: success
I20260129 14:56:12.292742 10076 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:56:12.299899 8507 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:56:12.304044 10021 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:50403; Asking this server to re-register.
I20260129 14:56:12.304783 8507 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:12.306493 10021 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:56:12.312513 8644 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:56:12.307720 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:12.313126 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:12.314172 10021 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:41585; Asking this server to re-register.
I20260129 14:56:12.314725 8644 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:12.316183 10021 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:56:12.454188 8779 heartbeater.cc:344] Connected to a master server at 127.7.171.252:43411
I20260129 14:56:12.455417 10021 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:40199; Asking this server to re-register.
I20260129 14:56:12.455928 8779 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:12.457093 10021 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:56:12.829474 10080 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:12.829699 10080 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:12.831410 10080 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:12.832360 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:12.833509 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:12.833673 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:12.833837 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:12.834126 10080 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 14:56:13.487242 10086 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:13.487443 10086 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:13.488087 10086 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:13.489971 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:13.490139 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:13.490304 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:13.490538 10086 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 14:56:14.214143 10091 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:14.214331 10091 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:14.214918 10091 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:14.216850 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:14.217046 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:14.217155 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:14.217337 10091 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 14:56:14.910450 10095 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:14.910612 10095 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:14.911238 10095 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:14.913105 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:14.913283 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:14.913408 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:14.913697 10095 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 14:56:15.231379 7855 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
/tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--webserver_interface=127.7.171.254
--webserver_port=33955
--builtin_ntp_servers=127.7.171.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260129 14:56:15.543946 10099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260129 14:56:15.544299 10099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260129 14:56:15.544415 10099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260129 14:56:15.552158 10099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260129 14:56:15.552248 10099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260129 14:56:15.552371 10099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260129 14:56:15.552433 10099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260129 14:56:15.561517 10099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.171.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40303
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.171.254:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.171.254
--webserver_port=33955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 29 Jan 2026 14:43:24 UTC on 24a791456cd2
build id 10248
ASAN enabled
I20260129 14:56:15.562932 10099 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 14:56:15.564618 10099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 14:56:15.571925 10104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:15.572042 10105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 14:56:15.572227 10107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 14:56:15.573309 10099 server_base.cc:1060] running on GCE node
I20260129 14:56:15.574046 10099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260129 14:56:15.575201 10099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260129 14:56:15.576386 10099 hybrid_clock.cc:648] HybridClock initialized: now 1769698575576352 us; error 46 us; skew 500 ppm
I20260129 14:56:15.576690 10099 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 14:56:15.578577 10099 webserver.cc:492] Webserver started at http://127.7.171.254:33955/ using document root <none> and password file <none>
I20260129 14:56:15.579205 10099 fs_manager.cc:362] Metadata directory not provided
I20260129 14:56:15.579301 10099 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 14:56:15.584281 10099 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260129 14:56:15.586701 10113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 14:56:15.587882 10099 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 14:56:15.588020 10099 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data,/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
uuid: "5cde2b69b78d4e8491572266f7019c6e"
format_stamp: "Formatted at 2026-01-29 14:55:43 on dist-test-slave-n0m6"
I20260129 14:56:15.588665 10099 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 14:56:15.632803 10099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 14:56:15.633612 10099 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 14:56:15.633881 10099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 14:56:15.653193 10099 rpc_server.cc:307] RPC server started. Bound to: 127.7.171.254:38695
I20260129 14:56:15.653228 10164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.171.254:38695 every 8 connection(s)
I20260129 14:56:15.654659 10099 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/master-0/data/info.pb
I20260129 14:56:15.658833 7855 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskf9Nfvs/build/asan/bin/kudu as pid 10099
I20260129 14:56:15.661252 10165 sys_catalog.cc:263] Verifying existing consensus state
I20260129 14:56:15.665055 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap starting.
I20260129 14:56:15.673838 10165 log.cc:826] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Log is configured to *not* fsync() on all Append() calls
I20260129 14:56:15.688342 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=16 ignored=0} inserts{seen=9 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260129 14:56:15.688896 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Bootstrap complete.
I20260129 14:56:15.693939 10165 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:15.696466 10165 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Initialized, Role: FOLLOWER
I20260129 14:56:15.697028 10165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 7.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:15.697765 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:15.697978 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:15.698769 10165 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 14:56:15.710965 10184 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:15.711059 10184 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: loading cluster ID for follower catalog manager: success
I20260129 14:56:15.715142 10184 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: acquiring CA information for follower catalog manager: success
I20260129 14:56:15.717820 10184 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 14:56:15.726207 10099 master_runner.cc:186] Error getting master registration for 127.7.171.253:34637: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111),
I20260129 14:56:15.729585 10099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:15.791805 8645 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:56:15.792297 8508 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:56:15.794440 10130 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0721a8132fc84354ae4b9e42fe387c67" instance_seqno: 1769698545794927) as {username='slave'} at 127.7.171.194:46539; Asking this server to re-register.
I20260129 14:56:15.794488 10129 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "824b258cd009461bb4f25beea2af6eb9" instance_seqno: 1769698545310488) as {username='slave'} at 127.7.171.193:54941; Asking this server to re-register.
I20260129 14:56:15.795064 8645 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:15.795173 8508 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:15.796506 10130 ts_manager.cc:194] Registered new tserver with Master: 0721a8132fc84354ae4b9e42fe387c67 (127.7.171.194:44263)
I20260129 14:56:15.796655 10129 ts_manager.cc:194] Registered new tserver with Master: 824b258cd009461bb4f25beea2af6eb9 (127.7.171.193:44151)
I20260129 14:56:16.079043 8780 heartbeater.cc:344] Connected to a master server at 127.7.171.254:38695
I20260129 14:56:16.080099 10129 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e6f177394443439dbabb0738a08b8fc3" instance_seqno: 1769698546235718) as {username='slave'} at 127.7.171.195:35345; Asking this server to re-register.
I20260129 14:56:16.080559 8780 heartbeater.cc:461] Registering TS with master...
I20260129 14:56:16.081617 10129 ts_manager.cc:194] Registered new tserver with Master: e6f177394443439dbabb0738a08b8fc3 (127.7.171.195:44395)
I20260129 14:56:16.271592 10190 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 14:56:16.271837 10190 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:16.273579 10190 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
W20260129 14:56:16.274497 10114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:16.276569 10114 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:16.280078 10031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" is_pre_election: true
I20260129 14:56:16.280447 10031 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 7.
I20260129 14:56:16.280959 10115 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:16.281276 10190 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 14:56:16.281425 10190 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 14:56:16.281533 10190 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 7 FOLLOWER]: Advancing to term 8
I20260129 14:56:16.284665 10190 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:16.285297 10190 leader_election.cc:290] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 8 election: Requested vote from peers c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637), 8db7ebac21af437fbdf8e1bd7e00fca2 (127.7.171.252:43411)
I20260129 14:56:16.285768 10031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cde2b69b78d4e8491572266f7019c6e" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2"
I20260129 14:56:16.285949 10031 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 7 FOLLOWER]: Advancing to term 8
W20260129 14:56:16.286320 10114 leader_election.cc:336] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:16.288823 10031 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cde2b69b78d4e8491572266f7019c6e in term 8.
I20260129 14:56:16.289176 10115 leader_election.cc:304] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cde2b69b78d4e8491572266f7019c6e, 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:16.289440 10190 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 8 FOLLOWER]: Leader election won for term 8
I20260129 14:56:16.289806 10190 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [term 8 LEADER]: Becoming Leader. State: Replica: 5cde2b69b78d4e8491572266f7019c6e, State: Running, Role: LEADER
I20260129 14:56:16.290201 10190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 7.19, Last appended by leader: 19, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:16.292430 10195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde2b69b78d4e8491572266f7019c6e. Latest consensus state: current_term: 8 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:16.292601 10195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: LEADER
W20260129 14:56:16.791834 10114 consensus_peers.cc:597] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e -> Peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Couldn't send request to peer c63d8c4ffa4047c9a8423771f4723d6e. Status: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 14:56:16.825227 10031 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Refusing update from remote peer 5cde2b69b78d4e8491572266f7019c6e: Log matching property violated. Preceding OpId in replica: term: 7 index: 19. Preceding OpId from leader: term: 8 index: 20. (index mismatch)
I20260129 14:56:16.826071 10195 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 16, Time since last communication: 0.000s
I20260129 14:56:16.828626 10202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde2b69b78d4e8491572266f7019c6e. Latest consensus state: current_term: 8 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:16.828866 10202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:16.834868 10195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:16.835165 10195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:56:16.835196 10202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:16.835371 10202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:16.835477 10195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:16.835594 10211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "5cde2b69b78d4e8491572266f7019c6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } } }
I20260129 14:56:16.835729 10195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e [sys.catalog]: This master's current role is: LEADER
I20260129 14:56:16.835809 10211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 14:56:16.938099 10197 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 14:56:16.940570 10197 catalog_manager.cc:679] Loaded metadata for table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed]
I20260129 14:56:16.940979 10197 catalog_manager.cc:679] Loaded metadata for table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc]
I20260129 14:56:16.942437 10197 tablet_loader.cc:96] loaded metadata for tablet 094f9f25d7d34067a81b8ae966f343d1 (table default.table_c2b80faabf21435d991d592f638440aa [id=908853f113ee48df891ae3a9db09c1ed])
I20260129 14:56:16.942742 10197 tablet_loader.cc:96] loaded metadata for tablet 0ad9a04affff45efaf91c5d9ac26ba26 (table default.table_6ecb7f939f3648ebab36e0f73528521a [id=b78f947e473549d9a57b0b87c8cc14fc])
I20260129 14:56:16.943635 10197 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 14:56:16.944433 10197 catalog_manager.cc:1269] Loaded cluster ID: 3338e861d4ab413a8a6faecf70450a0b
I20260129 14:56:16.944504 10197 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 14:56:16.945536 10197 catalog_manager.cc:1514] Loading token signing keys...
I20260129 14:56:16.946291 10197 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 5cde2b69b78d4e8491572266f7019c6e: Loaded TSK: 0
I20260129 14:56:16.947163 10197 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 14:56:16.948004 10197 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260129 14:56:16.948757 10197 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260129 14:56:17.029305 10130 master_service.cc:954] ReplaceTablet: received request to replace tablet 563c97415304403c8a50ff956fb54363 from {username='slave'} at 127.0.0.1:48802
I20260129 14:56:17.029839 10128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.030084 10127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_a0a36d2e9fd749c9958b59c392a0e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.030512 10121 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_acb782ad01664cc1a5a6ba4cb226d899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.030828 10123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_ffffe38dc46a4a97ba5f4b081cbcc822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.030979 10126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_95ece788510d4811b741bfafc484dd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.031435 10124 master_service.cc:954] ReplaceTablet: received request to replace tablet 740d94adef8b493bbbb88380b11c3404 from {username='slave'} at 127.0.0.1:48802
I20260129 14:56:17.031831 10129 master_service.cc:954] ReplaceTablet: received request to replace tablet 7b5bc5c45b2a4314b92c797aac0ca371 from {username='slave'} at 127.0.0.1:48802
I20260129 14:56:17.031844 10125 master_service.cc:954] ReplaceTablet: received request to replace tablet 563c97415304403c8a50ff956fb54363 from {username='slave'} at 127.0.0.1:48802
I20260129 14:56:17.031874 10122 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_b6e29b73e03042138b5a216f08dce0c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.032284 10130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_29883fd0b006400b953426f944f08202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.033147 10129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_663408aa31d3405ab53f9afa189ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.033185 10124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_17c52877671f4e30bcf4f682122e181e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 14:56:17.033524 10125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48802:
name: "default.Table_60640f05a979404380083e385f1e6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260129 14:56:17.037613 10183 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_c2b80faabf21435d991d592f638440aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2b80faabf21435d991d592f638440aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2b80faabf21435d991d592f638440aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_c2b80faabf21435d991d592f638440aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2b80faabf21435d991d592f638440aa\",\"kudu.table_id\":\"908853f113ee48df891ae3a9db09c1ed\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769698546, eventType=CREATE_TABLE, dbName=default, tableName=table_95ece788510d4811b741bfafc484dd89, message={"server":"","servicePrincipal":"","db":"default","table":"table_95ece788510d4811b741bfafc484dd89","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_95ece788510d4811b741bfafc484dd89\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769698546},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskf9Nfvs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1769698529561400-7855-0/minicluster-data/warehouse/table_95ece788510d4811b741bfafc484dd89\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_95ece788510d4811b741bfafc484dd89\",\"kudu.table_id\":\"66667881263c4bb0a181c0c7576354f7\",\"transient_lastDdlTime\":\"1769698546\",\"comment\":\"\",\"kudu.master_addresses\":\"127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"3338e861d4ab413a8a6faecf70450a0b\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769698546,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f9c11bd4dcd google::LogMessage::Fail() at ??:0
@ 0x7f9c11bd8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9c11bd47cc google::LogMessage::Flush() at ??:0
@ 0x7f9c11bd4a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9c285cb7f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9c285ca0a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f9c285d1d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f9c285d1b99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f9c14949f99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c17f566db start_thread at ??:0
@ 0x7f9c10a8571f clone at ??:0
W20260129 14:56:17.215965 8668 connection.cc:537] client connection to 127.7.171.254:38695 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260129 14:56:17.216928 8787 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.254:38695)
W20260129 14:56:17.217222 8787 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.171.254:38695)
W20260129 14:56:17.217942 8787 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.7.171.254:38695)
I20260129 14:56:17.314376 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:17.314539 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:17.331272 10211 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:17.331439 10211 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:17.332090 10211 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:17.334174 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:17.334527 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:17.334664 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:17.334921 10211 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:17.965541 10234 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:17.965725 10234 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:17.966266 10234 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:17.967136 10005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260129 14:56:17.968125 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:17.968262 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:17.968392 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:17.968595 10234 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:18.002432 7855 master-stress-test.cc:577] Killing test thread 1/25
I20260129 14:56:18.663126 10238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:18.663318 10238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:18.664014 10238 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:18.665818 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:18.665967 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:18.666091 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:18.666296 10238 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:19.741781 10243 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:19.741957 10243 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:19.742592 10243 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:19.744357 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:19.744498 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:19.744622 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:19.744885 10243 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:20.264984 10247 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:20.265185 10247 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:20.265940 10247 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:20.267676 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:20.267926 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:20.268062 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:20.268301 10247 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:21.872711 10251 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:21.872934 10251 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:21.873569 10251 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:21.875551 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:21.875978 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:21.876144 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:21.876360 10251 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:22.315478 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:22.315655 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:24.668645 10256 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:24.668844 10256 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:24.669442 10256 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:24.670420 10005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260129 14:56:24.671535 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:24.671710 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:24.671809 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:24.672010 10256 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:27.209158 10262 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:27.209358 10262 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:27.209967 10262 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:27.212163 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:27.212371 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:27.212494 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:27.212742 10262 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:27.316745 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:27.316947 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:31.976217 10271 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:31.976408 10271 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:31.977139 10271 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:31.978063 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260129 14:56:31.979447 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:31.979610 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:31.979768 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:31.980036 10271 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:32.317986 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:32.318186 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:35.062404 10277 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:35.062609 10277 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:35.063339 10277 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:35.065420 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:35.065580 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:35.065701 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:35.065905 10277 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:37.319366 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:37.319525 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:42.323379 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:42.323556 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260129 14:56:46.779120 8780 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (59 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:46.823601 8512 heartbeater.cc:646] Failed to heartbeat to 127.7.171.253:34637 (91 consecutive failures): Network error: Failed to ping master at 127.7.171.253:34637: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:46.849326 8646 heartbeater.cc:646] Failed to heartbeat to 127.7.171.253:34637 (91 consecutive failures): Network error: Failed to ping master at 127.7.171.253:34637: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:47.325230 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:47.325438 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:50.738601 10294 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:50.738796 10294 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:50.739640 10294 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:50.740725 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260129 14:56:50.741814 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:50.741964 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:50.742098 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:50.742342 10294 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:52.326833 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:52.327018 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:56.457268 10301 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:56.457482 10301 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:56.458096 10301 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:56.459095 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:56:56.460132 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:56.460284 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:56.460413 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:56.460656 10301 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:57.328694 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:56:57.328908 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:56:58.356544 10306 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:58.356711 10306 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:58.357225 10306 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:58.359164 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:56:58.359344 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:56:58.359470 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:58.359694 10306 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:56:59.856614 10310 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:56:59.856798 10310 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:56:59.857448 10310 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:56:59.859496 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:56:59.859687 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:56:59.859776 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:56:59.860021 10310 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:02.330400 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:02.330605 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:07.331918 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:07.332149 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:09.638790 10320 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:57:09.638983 10320 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:57:09.639642 10320 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:57:09.640748 10005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260129 14:57:09.641837 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:57:09.641986 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:57:09.642099 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:57:09.642313 10320 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:12.333611 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:12.333848 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:15.751888 10330 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:57:15.752110 10330 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:57:15.752751 10330 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:57:15.753796 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:57:15.755121 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:57:15.755283 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:57:15.755421 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:57:15.755633 10330 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:17.335335 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:17.335544 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:22.337109 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:22.337304 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:27.338721 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:27.338927 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:28.305068 10343 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:57:28.305266 10343 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:57:28.305925 10343 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:57:28.306758 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:57:28.308274 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:57:28.308404 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:57:28.308493 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:57:28.308715 10343 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:32.340415 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:32.340572 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:33.259492 10350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:57:33.259749 10350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:57:33.260380 10350 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:57:33.263914 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:57:33.264135 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:57:33.264266 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:57:33.264505 10350 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:37.341838 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:37.342013 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:39.187384 10357 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:57:39.187570 10357 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:57:39.188195 10357 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:57:39.189009 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260129 14:57:39.190093 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:57:39.190232 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:57:39.190331 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:57:39.190513 10357 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:41.337625 10362 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:57:41.337843 10362 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:57:41.338557 10362 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:57:41.342244 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:57:41.342447 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:57:41.342553 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:57:41.342823 10362 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:42.343288 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:42.343549 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260129 14:57:46.914666 8780 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (179 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:57:46.958626 8512 heartbeater.cc:646] Failed to heartbeat to 127.7.171.253:34637 (211 consecutive failures): Network error: Failed to ping master at 127.7.171.253:34637: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:57:46.980607 8645 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (179 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:57:47.345168 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:47.345353 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:49.215797 10394 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:57:49.216032 10394 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:57:49.217017 10394 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:57:49.217840 10005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260129 14:57:49.219777 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:57:49.219909 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:57:49.220018 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:57:49.220413 10394 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:57:52.346618 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:52.346802 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:57:57.348523 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:57:57.348680 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:02.350256 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:02.350425 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:02.602195 10465 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:58:02.602454 10465 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:58:02.603153 10465 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:58:02.603757 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:58:02.604925 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:58:02.605099 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:58:02.605193 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:58:02.605396 10465 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:58:07.351927 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:07.352187 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:12.353600 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:12.353794 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:12.367842 10010 maintenance_manager.cc:419] P 8db7ebac21af437fbdf8e1bd7e00fca2: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033360
I20260129 14:58:12.413069 10009 maintenance_manager.cc:643] P 8db7ebac21af437fbdf8e1bd7e00fca2: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.044s user 0.014s sys 0.010s Metrics: {"bytes_written":8122,"cfile_init":1,"dirs.queue_time_us":377,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":894,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":534,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":11,"thread_start_us":488,"threads_started":2,"wal-append.queue_time_us":342}
I20260129 14:58:12.414191 10010 maintenance_manager.cc:419] P 8db7ebac21af437fbdf8e1bd7e00fca2: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 714 bytes on disk
I20260129 14:58:12.414832 10009 maintenance_manager.cc:643] P 8db7ebac21af437fbdf8e1bd7e00fca2: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20260129 14:58:17.355391 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:17.355597 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:22.282043 10564 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:58:22.282256 10564 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:58:22.283157 10564 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:58:22.284114 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:58:22.285267 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:58:22.285600 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:58:22.285720 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:58:22.285940 10564 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:58:22.356751 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:22.356988 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:27.358487 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:27.358685 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:32.360554 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:32.360806 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:37.362390 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:37.362627 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:38.358429 10703 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:58:38.358657 10703 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:58:38.359535 10703 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:58:38.360702 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:58:38.361871 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:58:38.362018 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:58:38.362171 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:58:38.362445 10703 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:58:42.206080 10738 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:58:42.206324 10738 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:58:42.207026 10738 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:58:42.209321 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:58:42.209498 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:58:42.209658 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:58:42.209918 10738 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:58:42.364276 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:42.364521 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:43.115775 10749 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:58:43.115980 10749 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:58:43.116750 10749 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:58:43.118959 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:58:43.119179 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:58:43.119316 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:58:43.119551 10749 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260129 14:58:47.055622 8780 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (299 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:58:47.096753 8512 heartbeater.cc:646] Failed to heartbeat to 127.7.171.253:34637 (331 consecutive failures): Network error: Failed to ping master at 127.7.171.253:34637: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:58:47.157266 8645 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (299 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:58:47.366097 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:47.366317 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:51.514279 10825 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:58:51.514510 10825 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:58:51.515266 10825 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:58:51.516448 10005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260129 14:58:51.517750 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:58:51.517925 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:58:51.518062 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:58:51.518338 10825 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:58:52.368741 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:52.368965 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:58:56.873989 10874 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:58:56.874202 10874 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:58:56.874980 10874 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:58:56.876081 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:58:56.877295 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:58:56.877456 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:58:56.877595 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:58:56.877857 10874 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:58:57.370285 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:58:57.370508 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:02.372143 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:02.372344 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:03.913697 10936 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:59:03.913930 10936 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:59:03.915125 10936 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:59:03.916204 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:59:03.917641 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:59:03.917807 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:59:03.917949 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:59:03.918237 10936 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:59:07.374017 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:07.374215 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:12.375738 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:12.375967 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:17.377446 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:17.377635 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:22.378988 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:22.379170 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:23.737668 11093 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:59:23.737921 11093 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:59:23.738924 11093 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:59:23.740005 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 14:59:23.741637 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:59:23.741806 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:59:23.741974 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:59:23.742257 11093 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:59:26.516508 11121 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:59:26.516752 11121 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:59:26.517436 11121 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:59:26.519654 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:59:26.519800 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:59:26.519953 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:59:26.520223 11121 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:59:27.380456 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:27.380659 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:32.382154 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:32.382316 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:37.383919 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:37.384200 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:42.385542 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:42.385725 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:43.380139 11258 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:59:43.380326 11258 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:59:43.381054 11258 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:59:43.382037 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:59:43.382037 10005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260129 14:59:43.383145 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:59:43.383302 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:59:43.383397 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:59:43.383616 11258 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260129 14:59:47.197223 8780 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (419 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:59:47.237572 8512 heartbeater.cc:646] Failed to heartbeat to 127.7.171.253:34637 (451 consecutive failures): Network error: Failed to ping master at 127.7.171.253:34637: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:59:47.342509 8646 heartbeater.cc:646] Failed to heartbeat to 127.7.171.253:34637 (451 consecutive failures): Network error: Failed to ping master at 127.7.171.253:34637: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 14:59:47.387017 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:47.387233 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:52.388459 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:52.388634 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 14:59:55.410336 11375 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 14:59:55.410543 11375 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 14:59:55.411124 11375 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 14:59:55.412017 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 14:59:55.413058 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 14:59:55.413234 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 14:59:55.413370 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 14:59:55.413585 11375 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 14:59:57.389885 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 14:59:57.390048 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:02.391214 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:02.391379 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:07.392697 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:07.392877 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:09.034603 11540 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 15:00:09.034817 11540 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 15:00:09.035427 11540 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 15:00:09.036489 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 15:00:09.037537 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 15:00:09.037777 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 15:00:09.037873 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 15:00:09.038058 11540 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 15:00:12.394090 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:12.394250 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:13.496625 11596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 15:00:13.496831 11596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 15:00:13.497399 11596 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 15:00:13.499169 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 15:00:13.499329 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 15:00:13.499459 10005 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 15:00:13.499732 11596 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 15:00:17.395382 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:17.395546 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:21.645407 11671 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 15:00:21.645648 11671 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 15:00:21.646308 11671 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 15:00:21.647293 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260129 15:00:21.648514 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 15:00:21.648680 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 15:00:21.648809 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 15:00:21.649037 11671 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 15:00:22.396862 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:22.397205 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:27.398578 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:27.398789 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:32.401687 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:32.401866 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:34.507193 11820 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5cde2b69b78d4e8491572266f7019c6e)
I20260129 15:00:34.507381 11820 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde2b69b78d4e8491572266f7019c6e" member_type: VOTER last_known_addr { host: "127.7.171.254" port: 38695 } } peers { permanent_uuid: "c63d8c4ffa4047c9a8423771f4723d6e" member_type: VOTER last_known_addr { host: "127.7.171.253" port: 34637 } } peers { permanent_uuid: "8db7ebac21af437fbdf8e1bd7e00fca2" member_type: VOTER last_known_addr { host: "127.7.171.252" port: 43411 } }
I20260129 15:00:34.507957 11820 leader_election.cc:290] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695), c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637)
W20260129 15:00:34.508942 10006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260129 15:00:34.509951 10005 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c63d8c4ffa4047c9a8423771f4723d6e (127.7.171.253:34637): Network error: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
W20260129 15:00:34.510082 10006 leader_election.cc:336] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5cde2b69b78d4e8491572266f7019c6e (127.7.171.254:38695): Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
I20260129 15:00:34.510188 10006 leader_election.cc:304] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db7ebac21af437fbdf8e1bd7e00fca2; no voters: 5cde2b69b78d4e8491572266f7019c6e, c63d8c4ffa4047c9a8423771f4723d6e
I20260129 15:00:34.510386 11820 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8db7ebac21af437fbdf8e1bd7e00fca2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 15:00:37.403353 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:37.403544 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260129 15:00:42.404824 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:42.405017 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260129 15:00:47.322129 8780 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (539 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)
W20260129 15:00:47.395884 8787 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
W20260129 15:00:47.396188 8787 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
W20260129 15:00:47.396353 8787 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
W20260129 15:00:47.402413 8512 heartbeater.cc:646] Failed to heartbeat to 127.7.171.253:34637 (571 consecutive failures): Network error: Failed to ping master at 127.7.171.253:34637: Client connection negotiation failed: client connection to 127.7.171.253:34637: connect: Connection refused (error 111)
I20260129 15:00:47.406543 9990 master_runner.cc:186] Error getting master registration for 127.7.171.254:38695: Network error: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111),
I20260129 15:00:47.406751 9990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
F20260129 15:00:47.406929 8814 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
*** Aborted at 1769698847 (unix time) try "date -d @1769698847" if you are using GNU date ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411926 8795 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663408aa31d3405ab53f9afa189ceeaf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
W20260129 15:00:47.412901 8787 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.7.171.254:38695)
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411926 8795 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663408aa31d3405ab53f9afa189ceeaf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.413487 8800 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29883fd0b006400b953426f944f08202 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411926 8795 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663408aa31d3405ab53f9afa189ceeaf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.413487 8800 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29883fd0b006400b953426f944f08202 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414871 8794 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_acb782ad01664cc1a5a6ba4cb226d899 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411926 8795 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663408aa31d3405ab53f9afa189ceeaf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.413487 8800 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29883fd0b006400b953426f944f08202 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414871 8794 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_acb782ad01664cc1a5a6ba4cb226d899 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414939 8804 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411926 8795 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663408aa31d3405ab53f9afa189ceeaf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.413487 8800 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29883fd0b006400b953426f944f08202 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414871 8794 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_acb782ad01664cc1a5a6ba4cb226d899 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414939 8804 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414908 8803 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_95ece788510d4811b741bfafc484dd89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411926 8795 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663408aa31d3405ab53f9afa189ceeaf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.413487 8800 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29883fd0b006400b953426f944f08202 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414871 8794 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_acb782ad01664cc1a5a6ba4cb226d899 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414939 8804 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414908 8803 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_95ece788510d4811b741bfafc484dd89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414917 8799 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a0a36d2e9fd749c9958b59c392a0e64c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
F20260129 15:00:47.407274 8805 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.408594 8796 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20260129 15:00:47.408596 8798 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c2d5b6c10af64d2bb9e7531cc9d1b1c8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1) F20260129 15:00:47.410035 8797 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b6e29b73e03042138b5a216f08dce0c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411754 8802 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ffffe38dc46a4a97ba5f4b081cbcc822 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.411926 8795 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663408aa31d3405ab53f9afa189ceeaf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.413487 8800 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29883fd0b006400b953426f944f08202 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414871 8794 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_acb782ad01664cc1a5a6ba4cb226d899 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414939 8804 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414908 8803 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_95ece788510d4811b741bfafc484dd89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.414917 8799 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a0a36d2e9fd749c9958b59c392a0e64c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)F20260129 15:00:47.416013 8801 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_17c52877671f4e30bcf4f682122e181e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 731) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.7.171.254:38695,127.7.171.253:34637,127.7.171.252:43411, num_attempts: 1)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800001eaf) received by PID 7855 (TID 0x7fda53364700) from PID 7855; stack trace: ***
@ 0x7fda6ffb9980 (unknown) at ??:0
@ 0x7fda68d8cfb7 gsignal at ??:0
@ 0x7fda68d8e921 abort at ??:0
@ 0x7fda6bac4dcd google::LogMessage::Fail() at ??:0
@ 0x7fda6bac8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fda6bac47cc google::LogMessage::Flush() at ??:0
@ 0x7fda6bac5f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40d9f5 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:391
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fda694146df (unknown) at ??:0
@ 0x7fda6ffae6db start_thread at ??:0
@ 0x7fda68e6f71f clone at ??:0
W20260129 15:00:47.500726 8645 heartbeater.cc:646] Failed to heartbeat to 127.7.171.254:38695 (539 consecutive failures): Network error: Failed to ping master at 127.7.171.254:38695: Client connection negotiation failed: client connection to 127.7.171.254:38695: connect: Connection refused (error 111)