Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-05-02T02:02:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-02T02:02:24Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3_zYNP/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3_zYNP/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/security.properties
2026-05-02 02:02:35: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3_zYNP/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3_zYNP/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260502 02:02:47.322598 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.254:43423
--webserver_interface=127.2.168.254
--webserver_port=0
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 02:02:47.772241 3013 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:47.772928 3013 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:47.773073 3013 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:47.784489 3013 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:02:47.784643 3013 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:47.784713 3013 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:02:47.784754 3013 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:02:47.796782 3013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.254:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.168.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:47.799028 3013 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:47.801786 3013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:47.814538 3019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:47.814538 3021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:47.814647 3018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:47.814963 3013 server_base.cc:1061] running on GCE node
I20260502 02:02:47.816627 3013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:47.818495 3013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:47.819813 3013 hybrid_clock.cc:648] HybridClock initialized: now 1777687367819757 us; error 48 us; skew 500 ppm
I20260502 02:02:47.820369 3013 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:47.825102 3013 webserver.cc:492] Webserver started at http://127.2.168.254:44147/ using document root <none> and password file <none>
I20260502 02:02:47.826130 3013 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:47.826946 3013 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:47.827544 3013 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 02:02:47.831169 3013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/instance:
uuid: "7f40c77f3a794ac1b717b2a4400f34e7"
format_stamp: "Formatted at 2026-05-02 02:02:47 on dist-test-slave-n6ms"
I20260502 02:02:47.832563 3013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal/instance:
uuid: "7f40c77f3a794ac1b717b2a4400f34e7"
format_stamp: "Formatted at 2026-05-02 02:02:47 on dist-test-slave-n6ms"
I20260502 02:02:47.845268 3013 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.005s
I20260502 02:02:47.850739 3027 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:47.852978 3013 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260502 02:02:47.853339 3013 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
uuid: "7f40c77f3a794ac1b717b2a4400f34e7"
format_stamp: "Formatted at 2026-05-02 02:02:47 on dist-test-slave-n6ms"
I20260502 02:02:47.853606 3013 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:47.876385 3013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:47.879180 3013 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:47.879671 3013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:47.926807 3013 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.254:43423
I20260502 02:02:47.926851 3078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.254:43423 every 8 connection(s)
I20260502 02:02:47.929107 3013 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/info.pb
I20260502 02:02:47.932420 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3013
I20260502 02:02:47.932981 2723 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal/instance
I20260502 02:02:47.937728 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--webserver_interface=127.2.168.253
--webserver_port=0
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 02:02:47.940555 3079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 02:02:47.952960 3079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } has no permanent_uuid. Determining permanent_uuid...
I20260502 02:02:47.980417 3079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } has no permanent_uuid. Determining permanent_uuid...
W20260502 02:02:47.982285 3028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
W20260502 02:02:47.985195 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.253:44041: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
I20260502 02:02:48.034632 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } attempt: 1
W20260502 02:02:48.037807 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.253:44041: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
I20260502 02:02:48.106330 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } attempt: 2
W20260502 02:02:48.108948 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.253:44041: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
I20260502 02:02:48.200340 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } attempt: 3
W20260502 02:02:48.203395 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.253:44041: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
I20260502 02:02:48.347023 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } attempt: 4
W20260502 02:02:48.350099 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.253:44041: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
W20260502 02:02:48.417937 3080 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:48.418983 3080 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:48.419301 3080 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:48.430397 3080 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:02:48.430550 3080 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:48.430668 3080 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:02:48.430747 3080 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:02:48.443900 3080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.168.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:48.445823 3080 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:48.447942 3080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:48.459987 3089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:48.461041 3091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:48.461440 3088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:48.462499 3080 server_base.cc:1061] running on GCE node
I20260502 02:02:48.463712 3080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:48.465894 3080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:48.467808 3080 hybrid_clock.cc:648] HybridClock initialized: now 1777687368467763 us; error 102 us; skew 500 ppm
I20260502 02:02:48.468508 3080 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:48.471808 3080 webserver.cc:492] Webserver started at http://127.2.168.253:33809/ using document root <none> and password file <none>
I20260502 02:02:48.472752 3080 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:48.472983 3080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:48.473412 3080 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 02:02:48.476280 3080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/instance:
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:02:48.477941 3080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal/instance:
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:02:48.486019 3080 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260502 02:02:48.491842 3097 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:48.494861 3080 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260502 02:02:48.495275 3080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:02:48.495764 3080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:48.514108 3080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:48.515157 3080 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:48.515498 3080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:48.542250 3080 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.253:44041
I20260502 02:02:48.542265 3148 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.253:44041 every 8 connection(s)
I20260502 02:02:48.544373 3080 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
I20260502 02:02:48.549800 3149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 02:02:48.550307 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3080
I20260502 02:02:48.550547 2723 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal/instance
I20260502 02:02:48.551710 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.252:41437
--webserver_interface=127.2.168.252
--webserver_port=0
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 02:02:48.556762 3149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } has no permanent_uuid. Determining permanent_uuid...
I20260502 02:02:48.591588 3149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } has no permanent_uuid. Determining permanent_uuid...
I20260502 02:02:48.603130 3149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } has no permanent_uuid. Determining permanent_uuid...
W20260502 02:02:48.604557 3100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:02:48.607082 3149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:48.649536 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } attempt: 5
I20260502 02:02:48.656618 3149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 1
I20260502 02:02:48.657997 3079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } has no permanent_uuid. Determining permanent_uuid...
W20260502 02:02:48.659178 3149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:02:48.660429 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:48.711792 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 1
W20260502 02:02:48.714262 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:48.727530 3149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 2
W20260502 02:02:48.729761 3149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:48.782686 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 2
W20260502 02:02:48.785296 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:48.821087 3149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 3
W20260502 02:02:48.823616 3149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:48.891729 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 3
W20260502 02:02:48.895166 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:48.966943 3149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 4
W20260502 02:02:48.969707 3149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:02:48.988914 3150 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:48.989300 3150 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:48.989388 3150 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:49.003298 3150 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:02:49.003571 3150 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:49.003661 3150 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:02:49.003708 3150 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:02:49.018132 3150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.252:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.168.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:49.020979 3150 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:49.023663 3150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:49.035645 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:49.035125 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:49.035892 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:49.038363 3150 server_base.cc:1061] running on GCE node
I20260502 02:02:49.039346 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:49.041332 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:49.042637 3150 hybrid_clock.cc:648] HybridClock initialized: now 1777687369042553 us; error 99 us; skew 500 ppm
I20260502 02:02:49.043140 3150 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:49.046360 3150 webserver.cc:492] Webserver started at http://127.2.168.252:38947/ using document root <none> and password file <none>
I20260502 02:02:49.047372 3150 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:49.047537 3150 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:49.048522 3150 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 02:02:49.052249 3150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/instance:
uuid: "32ce575a90d74bd588ce4dd4930dc6ef"
format_stamp: "Formatted at 2026-05-02 02:02:49 on dist-test-slave-n6ms"
I20260502 02:02:49.053324 3150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal/instance:
uuid: "32ce575a90d74bd588ce4dd4930dc6ef"
format_stamp: "Formatted at 2026-05-02 02:02:49 on dist-test-slave-n6ms"
I20260502 02:02:49.060158 3150 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260502 02:02:49.065776 3168 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:49.068068 3150 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260502 02:02:49.068403 3150 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
uuid: "32ce575a90d74bd588ce4dd4930dc6ef"
format_stamp: "Formatted at 2026-05-02 02:02:49 on dist-test-slave-n6ms"
I20260502 02:02:49.068673 3150 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:49.072518 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 4
W20260502 02:02:49.075637 3079 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:49.092559 3150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:49.093755 3150 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:49.094054 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:49.125187 3219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.252:41437 every 8 connection(s)
I20260502 02:02:49.125207 3150 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.252:41437
I20260502 02:02:49.127322 3150 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
I20260502 02:02:49.128198 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3150
I20260502 02:02:49.128561 2723 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal/instance
I20260502 02:02:49.133563 3220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 02:02:49.146920 3220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } has no permanent_uuid. Determining permanent_uuid...
I20260502 02:02:49.173416 3220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } has no permanent_uuid. Determining permanent_uuid...
I20260502 02:02:49.185184 3220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } has no permanent_uuid. Determining permanent_uuid...
I20260502 02:02:49.205469 3220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Bootstrap starting.
I20260502 02:02:49.211568 3220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Neither blocks nor log segments found. Creating new log.
I20260502 02:02:49.213155 3220 log.cc:826] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Log is configured to *not* fsync() on all Append() calls
I20260502 02:02:49.217198 3220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: No bootstrap required, opened a new log
I20260502 02:02:49.224851 3220 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.225824 3220 raft_consensus.cc:385] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 02:02:49.225983 3220 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Initialized, Role: FOLLOWER
I20260502 02:02:49.227032 3220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.228325 3227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.228683 3227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:49.229931 3220 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: configured and running, proceeding with master startup.
W20260502 02:02:49.246794 3241 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260502 02:02:49.247097 3241 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260502 02:02:49.242146 3150 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: OK, instance_id { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" instance_seqno: 1777687367913542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260502 02:02:49.249809 3150 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:02:49.268976 3149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 5
I20260502 02:02:49.294945 3149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap starting.
I20260502 02:02:49.303407 3149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Neither blocks nor log segments found. Creating new log.
I20260502 02:02:49.305419 3149 log.cc:826] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Log is configured to *not* fsync() on all Append() calls
I20260502 02:02:49.320286 3149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: No bootstrap required, opened a new log
I20260502 02:02:49.333532 3149 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.334092 3149 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 02:02:49.334367 3149 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6641b5408d2947bdb1a7d508a61f17e0, State: Initialized, Role: FOLLOWER
I20260502 02:02:49.335484 3149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
W20260502 02:02:49.348227 3174 tablet.cc:2404] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260502 02:02:49.352670 3149 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:02:49.352898 3079 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } attempt: 5
I20260502 02:02:49.354915 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.355966 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:49.371487 3079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Bootstrap starting.
I20260502 02:02:49.389645 3079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Neither blocks nor log segments found. Creating new log.
I20260502 02:02:49.391928 3079 log.cc:826] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Log is configured to *not* fsync() on all Append() calls
I20260502 02:02:49.404583 3079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: No bootstrap required, opened a new log
I20260502 02:02:49.413424 3079 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.413972 3079 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 02:02:49.414155 3079 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f40c77f3a794ac1b717b2a4400f34e7, State: Initialized, Role: FOLLOWER
I20260502 02:02:49.415247 3079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.423432 3258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.425379 3079 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:02:49.426033 3258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:49.428540 3227 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:02:49.428861 3227 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.430739 3227 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
I20260502 02:02:49.431710 3054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" is_pre_election: true
I20260502 02:02:49.432272 3054 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 0.
I20260502 02:02:49.433210 3124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:02:49.433853 3124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 0.
I20260502 02:02:49.434784 3171 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7; no voters:
I20260502 02:02:49.436719 3227 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260502 02:02:49.436944 3227 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 02:02:49.437052 3227 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 0 FOLLOWER]: Advancing to term 1
I20260502 02:02:49.441501 3227 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.443073 3054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f40c77f3a794ac1b717b2a4400f34e7"
I20260502 02:02:49.443332 3054 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 0 FOLLOWER]: Advancing to term 1
I20260502 02:02:49.446771 3054 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 1.
I20260502 02:02:49.447533 3171 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7; no voters:
I20260502 02:02:49.450038 3124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0"
I20260502 02:02:49.450296 3124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 0 FOLLOWER]: Advancing to term 1
I20260502 02:02:49.457844 3124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 1.
I20260502 02:02:49.451472 3080 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: OK, instance_id { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" instance_seqno: 1777687367913542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260502 02:02:49.461630 3227 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 1 election: Requested vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
I20260502 02:02:49.462239 3227 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260502 02:02:49.462416 3227 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 1 LEADER]: Becoming Leader. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Running, Role: LEADER
I20260502 02:02:49.462849 3080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:02:49.463845 3227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:49.485096 3265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce575a90d74bd588ce4dd4930dc6ef. Latest consensus state: current_term: 1 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.485702 3265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:49.491521 3268 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 02:02:49.494833 3268 catalog_manager.cc:1494] Initializing Kudu cluster ID...
W20260502 02:02:49.508019 3264 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260502 02:02:49.508703 3264 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260502 02:02:49.519164 3103 tablet.cc:2404] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260502 02:02:49.521035 3124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 1 FOLLOWER]: Refusing update from remote peer 32ce575a90d74bd588ce4dd4930dc6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260502 02:02:49.522328 3265 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260502 02:02:49.530746 3054 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Refusing update from remote peer 32ce575a90d74bd588ce4dd4930dc6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260502 02:02:49.535625 3265 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260502 02:02:49.585811 3258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce575a90d74bd588ce4dd4930dc6ef. Latest consensus state: current_term: 1 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.586125 3258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:49.590171 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce575a90d74bd588ce4dd4930dc6ef. Latest consensus state: current_term: 1 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.590580 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:49.590776 3271 mvcc.cc:204] Tried to move back new op lower bound from 7281407465543065600 to 7281407465329577984. Current Snapshot: MvccSnapshot[applied={T|T < 7281407465543065600}]
I20260502 02:02:49.596858 3275 mvcc.cc:204] Tried to move back new op lower bound from 7281407465543065600 to 7281407465329577984. Current Snapshot: MvccSnapshot[applied={T|T < 7281407465543065600}]
I20260502 02:02:49.601125 3227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.601425 3227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:49.601826 3227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.602010 3227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:49.602370 3274 mvcc.cc:204] Tried to move back new op lower bound from 7281407465543065600 to 7281407465329577984. Current Snapshot: MvccSnapshot[applied={T|T < 7281407465543065600}]
I20260502 02:02:49.611758 3258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.612110 3258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:49.617974 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:49.618214 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:49.621649 3268 catalog_manager.cc:1357] Generated new cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:49.621822 3268 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 02:02:49.653618 3268 catalog_manager.cc:1380] Generated new certificate authority record
I20260502 02:02:49.655704 3268 catalog_manager.cc:1514] Loading token signing keys...
I20260502 02:02:49.687525 3268 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Generated new TSK 0
I20260502 02:02:49.688859 3268 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 02:02:49.690615 3268 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 02:02:49.692056 3268 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 02:02:49.714969 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.168.193:0
--local_ip_for_outbound_sockets=127.2.168.193
--webserver_interface=127.2.168.193
--webserver_port=0
--tserver_master_addrs=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260502 02:02:49.725309 3291 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260502 02:02:50.201308 3293 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32ce575a90d74bd588ce4dd4930dc6ef)
I20260502 02:02:50.201598 3293 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:50.203178 3293 leader_election.cc:290] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:50.203879 3123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:02:50.204188 3195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" is_pre_election: true
I20260502 02:02:50.205120 3030 leader_election.cc:304] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 6641b5408d2947bdb1a7d508a61f17e0
I20260502 02:02:50.205596 3293 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260502 02:02:50.275219 3292 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:50.275684 3292 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:50.275923 3292 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:50.287350 3292 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:50.287607 3292 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.168.193
I20260502 02:02:50.299273 3292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.168.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.168.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.168.193
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:50.301271 3292 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:50.303347 3292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:50.314522 3299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:50.314563 3298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:50.314599 3301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:50.316030 3292 server_base.cc:1061] running on GCE node
I20260502 02:02:50.316946 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:50.318004 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:50.319265 3292 hybrid_clock.cc:648] HybridClock initialized: now 1777687370319163 us; error 67 us; skew 500 ppm
I20260502 02:02:50.319757 3292 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:50.322268 3292 webserver.cc:492] Webserver started at http://127.2.168.193:45575/ using document root <none> and password file <none>
I20260502 02:02:50.323153 3292 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:50.323266 3292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:50.323956 3292 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 02:02:50.326994 3292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data/instance:
uuid: "13f4d60cbae24d7681fd5a3800b323d0"
format_stamp: "Formatted at 2026-05-02 02:02:50 on dist-test-slave-n6ms"
I20260502 02:02:50.327893 3292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/wal/instance:
uuid: "13f4d60cbae24d7681fd5a3800b323d0"
format_stamp: "Formatted at 2026-05-02 02:02:50 on dist-test-slave-n6ms"
I20260502 02:02:50.335052 3292 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260502 02:02:50.339982 3307 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:50.342386 3292 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260502 02:02:50.342625 3292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/wal
uuid: "13f4d60cbae24d7681fd5a3800b323d0"
format_stamp: "Formatted at 2026-05-02 02:02:50 on dist-test-slave-n6ms"
I20260502 02:02:50.342878 3292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:50.362944 3292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:50.364108 3292 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:50.364450 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:50.365789 3292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260502 02:02:50.368038 3292 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260502 02:02:50.368227 3292 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:50.368368 3292 ts_tablet_manager.cc:616] Registered 0 tablets
I20260502 02:02:50.368585 3292 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:50.409427 3292 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.193:33145
I20260502 02:02:50.409472 3419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.193:33145 every 8 connection(s)
I20260502 02:02:50.411126 3292 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/data/info.pb
I20260502 02:02:50.414014 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3292
I20260502 02:02:50.414307 2723 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-0/wal/instance
I20260502 02:02:50.421510 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.168.194:0
--local_ip_for_outbound_sockets=127.2.168.194
--webserver_interface=127.2.168.194
--webserver_port=0
--tserver_master_addrs=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260502 02:02:50.439338 3420 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:02:50.440057 3420 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:50.442086 3420 heartbeater.cc:507] Master 127.2.168.252:41437 requested a full tablet report, sending...
I20260502 02:02:50.445091 3185 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:02:50.447175 3185 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.168.193:41467
I20260502 02:02:50.449220 3421 heartbeater.cc:344] Connected to a master server at 127.2.168.254:43423
I20260502 02:02:50.449435 3421 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:50.450349 3421 heartbeater.cc:507] Master 127.2.168.254:43423 requested a full tablet report, sending...
I20260502 02:02:50.450531 3422 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:02:50.450716 3422 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:50.451143 3422 heartbeater.cc:507] Master 127.2.168.253:44041 requested a full tablet report, sending...
I20260502 02:02:50.452435 3044 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:02:50.453102 3114 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:02:50.511008 3264 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:50.511139 3264 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: loading cluster ID for follower catalog manager: success
I20260502 02:02:50.514514 3264 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: acquiring CA information for follower catalog manager: success
I20260502 02:02:50.516682 3264 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:02:50.732393 3291 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:50.732564 3291 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: loading cluster ID for follower catalog manager: success
I20260502 02:02:50.737423 3291 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: acquiring CA information for follower catalog manager: success
I20260502 02:02:50.740654 3291 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260502 02:02:50.820676 3430 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:50.821256 3430 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:50.821416 3430 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:50.831806 3430 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:50.832059 3430 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.168.194
I20260502 02:02:50.844373 3430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.168.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.168.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.168.194
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:50.846273 3430 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:50.848766 3430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:50.863418 3437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:50.864485 3436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:50.870786 3439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:50.871858 3430 server_base.cc:1061] running on GCE node
I20260502 02:02:50.872881 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:50.874393 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:50.875695 3430 hybrid_clock.cc:648] HybridClock initialized: now 1777687370875570 us; error 132 us; skew 500 ppm
I20260502 02:02:50.876137 3430 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:50.879583 3430 webserver.cc:492] Webserver started at http://127.2.168.194:33449/ using document root <none> and password file <none>
I20260502 02:02:50.880800 3430 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:50.881009 3430 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:50.881472 3430 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 02:02:50.885111 3430 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data/instance:
uuid: "6b5c908dbe834186ab31f375cc37931a"
format_stamp: "Formatted at 2026-05-02 02:02:50 on dist-test-slave-n6ms"
I20260502 02:02:50.886301 3430 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/wal/instance:
uuid: "6b5c908dbe834186ab31f375cc37931a"
format_stamp: "Formatted at 2026-05-02 02:02:50 on dist-test-slave-n6ms"
I20260502 02:02:50.895087 3430 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260502 02:02:50.900450 3445 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:50.902755 3430 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260502 02:02:50.903015 3430 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/wal
uuid: "6b5c908dbe834186ab31f375cc37931a"
format_stamp: "Formatted at 2026-05-02 02:02:50 on dist-test-slave-n6ms"
I20260502 02:02:50.903379 3430 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:50.929593 3430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:50.930886 3430 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:50.931262 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:50.932766 3430 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260502 02:02:50.935820 3430 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260502 02:02:50.935968 3430 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:50.936169 3430 ts_tablet_manager.cc:616] Registered 0 tablets
I20260502 02:02:50.936257 3430 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:50.954056 3420 heartbeater.cc:499] Master 127.2.168.252:41437 was elected leader, sending a full tablet report...
I20260502 02:02:51.008948 3430 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.194:38029
I20260502 02:02:51.008960 3557 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.194:38029 every 8 connection(s)
I20260502 02:02:51.010597 3430 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/data/info.pb
I20260502 02:02:51.017249 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3430
I20260502 02:02:51.017578 2723 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-1/wal/instance
I20260502 02:02:51.027781 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.168.195:0
--local_ip_for_outbound_sockets=127.2.168.195
--webserver_interface=127.2.168.195
--webserver_port=0
--tserver_master_addrs=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260502 02:02:51.054365 3559 heartbeater.cc:344] Connected to a master server at 127.2.168.254:43423
I20260502 02:02:51.055008 3559 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:51.056037 3559 heartbeater.cc:507] Master 127.2.168.254:43423 requested a full tablet report, sending...
I20260502 02:02:51.054373 3560 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:02:51.057103 3560 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:51.057595 3560 heartbeater.cc:507] Master 127.2.168.253:44041 requested a full tablet report, sending...
I20260502 02:02:51.058306 3558 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:02:51.058446 3558 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:51.058918 3558 heartbeater.cc:507] Master 127.2.168.252:41437 requested a full tablet report, sending...
I20260502 02:02:51.059077 3114 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:02:51.060225 3044 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:02:51.062848 3185 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:02:51.063735 3185 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.168.194:47075
I20260502 02:02:51.245502 3569 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32ce575a90d74bd588ce4dd4930dc6ef)
I20260502 02:02:51.245770 3569 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:51.247038 3569 leader_election.cc:290] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:51.247450 3195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" is_pre_election: true
I20260502 02:02:51.247722 3123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:02:51.248739 3028 leader_election.cc:304] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 6641b5408d2947bdb1a7d508a61f17e0
I20260502 02:02:51.249172 3569 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260502 02:02:51.544816 3568 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:51.545665 3568 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:51.546813 3568 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:51.562749 3568 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:51.563103 3568 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.168.195
I20260502 02:02:51.567483 3558 heartbeater.cc:499] Master 127.2.168.252:41437 was elected leader, sending a full tablet report...
I20260502 02:02:51.585309 3568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.168.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.168.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.168.195
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:51.590409 3568 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:51.596267 3568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:51.616380 3575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:51.616451 3574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:51.616648 3577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:51.619415 3568 server_base.cc:1061] running on GCE node
I20260502 02:02:51.620517 3568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:51.623701 3568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:51.625371 3568 hybrid_clock.cc:648] HybridClock initialized: now 1777687371625302 us; error 109 us; skew 500 ppm
I20260502 02:02:51.625934 3568 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:51.629169 3568 webserver.cc:492] Webserver started at http://127.2.168.195:35267/ using document root <none> and password file <none>
I20260502 02:02:51.630215 3568 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:51.630339 3568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:51.630749 3568 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 02:02:51.635892 3568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data/instance:
uuid: "579c61089da24972849347b57060a11f"
format_stamp: "Formatted at 2026-05-02 02:02:51 on dist-test-slave-n6ms"
I20260502 02:02:51.637372 3568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/wal/instance:
uuid: "579c61089da24972849347b57060a11f"
format_stamp: "Formatted at 2026-05-02 02:02:51 on dist-test-slave-n6ms"
I20260502 02:02:51.647408 3568 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260502 02:02:51.653261 3583 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:51.656935 3568 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260502 02:02:51.657387 3568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/wal
uuid: "579c61089da24972849347b57060a11f"
format_stamp: "Formatted at 2026-05-02 02:02:51 on dist-test-slave-n6ms"
I20260502 02:02:51.657866 3568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:51.684708 3568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:51.686061 3568 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:51.686499 3568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:51.687736 3568 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260502 02:02:51.691079 3568 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260502 02:02:51.691237 3568 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:51.691699 3568 ts_tablet_manager.cc:616] Registered 0 tablets
I20260502 02:02:51.691808 3568 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:51.751910 3568 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.195:39731
I20260502 02:02:51.751943 3695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.195:39731 every 8 connection(s)
I20260502 02:02:51.756345 3568 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/data/info.pb
I20260502 02:02:51.764670 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3568
I20260502 02:02:51.764912 2723 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/ts-2/wal/instance
I20260502 02:02:51.792790 3697 heartbeater.cc:344] Connected to a master server at 127.2.168.254:43423
I20260502 02:02:51.793733 3698 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:02:51.793756 3697 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:51.794498 3698 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:51.797031 3696 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:02:51.797323 3696 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:51.798529 3697 heartbeater.cc:507] Master 127.2.168.254:43423 requested a full tablet report, sending...
I20260502 02:02:51.799018 3696 heartbeater.cc:507] Master 127.2.168.252:41437 requested a full tablet report, sending...
I20260502 02:02:51.800611 3698 heartbeater.cc:507] Master 127.2.168.253:44041 requested a full tablet report, sending...
I20260502 02:02:51.801908 3044 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:02:51.802408 3114 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:02:51.804075 3185 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:02:51.805452 3185 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.168.195:57905
I20260502 02:02:51.815640 2723 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260502 02:02:51.849156 3713 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32ce575a90d74bd588ce4dd4930dc6ef)
I20260502 02:02:51.849399 3713 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:51.850800 3713 leader_election.cc:290] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:51.851279 3195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" is_pre_election: true
I20260502 02:02:51.851320 3123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:02:51.852169 3028 leader_election.cc:304] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 6641b5408d2947bdb1a7d508a61f17e0
I20260502 02:02:51.852598 3713 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260502 02:02:51.852649 3714 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32ce575a90d74bd588ce4dd4930dc6ef)
I20260502 02:02:51.852901 3714 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:51.855288 3714 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:51.856745 3195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6641b5408d2947bdb1a7d508a61f17e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" is_pre_election: true
I20260502 02:02:51.856801 3053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6641b5408d2947bdb1a7d508a61f17e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" is_pre_election: true
I20260502 02:02:51.857210 3053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6641b5408d2947bdb1a7d508a61f17e0 in term 1.
I20260502 02:02:51.857928 3100 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0, 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef
I20260502 02:02:51.858682 3714 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260502 02:02:51.859391 3714 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32ce575a90d74bd588ce4dd4930dc6ef)
I20260502 02:02:51.859923 3714 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 1 FOLLOWER]: Advancing to term 2
I20260502 02:02:51.862502 3714 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:51.864679 3714 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 2 election: Requested vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:51.865783 3053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6641b5408d2947bdb1a7d508a61f17e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7f40c77f3a794ac1b717b2a4400f34e7"
I20260502 02:02:51.866089 3195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6641b5408d2947bdb1a7d508a61f17e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32ce575a90d74bd588ce4dd4930dc6ef"
I20260502 02:02:51.866053 3053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 1 FOLLOWER]: Advancing to term 2
I20260502 02:02:51.870491 3053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6641b5408d2947bdb1a7d508a61f17e0 in term 2.
I20260502 02:02:51.871459 3100 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0, 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef
I20260502 02:02:51.872406 3714 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260502 02:02:51.877883 3185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_90436a402bba47a3a912ef82a27459b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.879276 3184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_a61cfc1553f64513ad3755c97eebe4b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.881045 3714 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 LEADER]: Becoming Leader. State: Replica: 6641b5408d2947bdb1a7d508a61f17e0, State: Running, Role: LEADER
I20260502 02:02:51.881881 3714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:51.886824 3123 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 LEADER]: Rejecting Update request from peer 32ce575a90d74bd588ce4dd4930dc6ef for earlier term 1. Current term is 2. Ops: []
I20260502 02:02:51.886967 3183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_1377e577a29b415a8c3c80e6b9d577a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.888130 3719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6641b5408d2947bdb1a7d508a61f17e0. Latest consensus state: current_term: 2 leader_uuid: "6641b5408d2947bdb1a7d508a61f17e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:51.889183 3719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:51.889288 3718 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260502 02:02:51.896488 3723 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260502 02:02:51.899179 3185 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_90436a402bba47a3a912ef82a27459b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260502 02:02:51.902707 3718 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Peer 6641b5408d2947bdb1a7d508a61f17e0 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260502 02:02:51.902864 3723 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 02:02:51.903635 3724 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 1 LEADER]: Stepping down as leader of term 1
I20260502 02:02:51.903893 3724 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Running, Role: LEADER
I20260502 02:02:51.904395 3723 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:51.904690 3723 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 02:02:51.904770 3724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:51.905735 3724 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 1 FOLLOWER]: Advancing to term 2
I20260502 02:02:51.906723 3723 catalog_manager.cc:1514] Loading token signing keys...
I20260502 02:02:51.908072 3723 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Loaded TSK: 0
I20260502 02:02:51.911682 3723 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 02:02:51.913540 3723 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 02:02:51.915201 3723 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
W20260502 02:02:51.928983 3708 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.168.252:41437)
I20260502 02:02:51.934859 3114 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57796:
name: "default.Table_6020a1239f724c2bb65f4e894b46745b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.943025 3113 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57796:
name: "default.Table_8ddc8addf29b495d8eb6b6e9f95577f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.943912 3112 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57796:
name: "default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260502 02:02:51.950169 3114 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6020a1239f724c2bb65f4e894b46745b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260502 02:02:51.950089 3111 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57796:
name: "default.Table_11d590196f20472e88078f6174a018fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.959918 3110 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57796:
name: "default.Table_e82eadcb627540889fd970fec3f1bb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.969838 3109 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57796:
name: "default.Table_b795fd960741406aab13f8b310fe8812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:51.972918 3108 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57796:
name: "default.Table_15a6efb8f407476ba8bcc483a784f4c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:52.053587 2723 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task3_zYNP/build/asan/bin/kudu pid 3080
I20260502 02:02:52.055486 3144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260502 02:02:52.252964 3241 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:02:52.262080 2723 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3_zYNP/build/asan/bin/kudu with pid 3080
W20260502 02:02:52.304092 3560 heartbeater.cc:646] Failed to heartbeat to 127.2.168.253:44041 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.2.168.253:44041 (error 108)
I20260502 02:02:52.309127 3696 heartbeater.cc:499] Master 127.2.168.252:41437 was elected leader, sending a full tablet report...
W20260502 02:02:52.317013 3712 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 02:02:52.300026 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.168.253:44041 (local address 127.0.0.1:57802)
0502 02:02:52.300455 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0502 02:02:52.300479 (+ 24us) client_negotiation.cc:175] Beginning negotiation
0502 02:02:52.300852 (+ 373us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0502 02:02:52.315511 (+ 14659us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141}
W20260502 02:02:52.319334 3753 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 02:02:52.309108 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.168.253:44041 (local address 127.2.168.195:59265)
0502 02:02:52.318042 (+ 8934us) negotiation.cc:107] Waiting for socket to connect
0502 02:02:52.318222 (+ 180us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":8823,"thread_start_us":8706,"threads_started":1}
W20260502 02:02:52.320015 3698 heartbeater.cc:646] Failed to heartbeat to 127.2.168.253:44041 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection reset by peer (error 104)
I20260502 02:02:52.461918 3754 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:02:52.462162 3754 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:52.463344 3754 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
I20260502 02:02:52.463943 3053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" is_pre_election: true
I20260502 02:02:52.464413 3053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 2.
I20260502 02:02:52.465130 3171 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7; no voters:
W20260502 02:02:52.465189 3169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
I20260502 02:02:52.465318 3422 heartbeater.cc:499] Master 127.2.168.253:44041 was elected leader, sending a full tablet report...
I20260502 02:02:52.465751 3754 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260502 02:02:52.465986 3754 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 02:02:52.466216 3754 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 2 FOLLOWER]: Advancing to term 3
W20260502 02:02:52.467411 3169 leader_election.cc:336] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041): Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
W20260502 02:02:52.468451 3422 heartbeater.cc:646] Failed to heartbeat to 127.2.168.253:44041 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
I20260502 02:02:52.482991 3754 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:52.484483 3754 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 3 election: Requested vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
I20260502 02:02:52.484746 3053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7f40c77f3a794ac1b717b2a4400f34e7"
I20260502 02:02:52.485047 3053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 2 FOLLOWER]: Advancing to term 3
W20260502 02:02:52.487039 3169 leader_election.cc:336] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041): Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111)
I20260502 02:02:52.584542 3053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 3.
I20260502 02:02:52.589569 3171 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 6641b5408d2947bdb1a7d508a61f17e0
I20260502 02:02:52.590243 3754 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 FOLLOWER]: Leader election won for term 3
I20260502 02:02:52.590982 3754 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 LEADER]: Becoming Leader. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Running, Role: LEADER
I20260502 02:02:52.591506 3754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:52.602387 3759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce575a90d74bd588ce4dd4930dc6ef. Latest consensus state: current_term: 3 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:52.602959 3759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:52.828702 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--webserver_interface=127.2.168.253
--webserver_port=33809
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 02:02:53.027902 3053 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 3 FOLLOWER]: Refusing update from remote peer 32ce575a90d74bd588ce4dd4930dc6ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260502 02:02:53.028793 3759 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260502 02:02:53.043591 3169 consensus_peers.cc:597] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef -> Peer 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041): Couldn't send request to peer 6641b5408d2947bdb1a7d508a61f17e0. Status: Network error: Client connection negotiation failed: client connection to 127.2.168.253:44041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260502 02:02:53.074131 3754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:53.074483 3754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:53.087699 3768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce575a90d74bd588ce4dd4930dc6ef. Latest consensus state: current_term: 3 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:53.088124 3768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:53.088264 3773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:53.088490 3773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260502 02:02:53.226766 3238 hms_client.cc:285] Time spent create HMS table: real 1.306s user 0.001s sys 0.000s
I20260502 02:02:53.227471 3185 catalog_manager.cc:2174] created HMS catalog entry for table [id=4d8c0f498fbf4a849444ecf94fa8f3eb]
I20260502 02:02:53.240661 3185 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53004 (request call id 1) took 1363 ms. Trace:
I20260502 02:02:53.241227 3185 rpcz_store.cc:276] 0502 02:02:51.876751 (+ 0us) service_pool.cc:167] Inserting onto call queue
0502 02:02:51.876877 (+ 126us) service_pool.cc:224] Handling call
0502 02:02:51.899915 (+ 23038us) catalog_manager.cc:2101] Acquired catalog manager lock
0502 02:02:51.904995 (+ 5080us) catalog_manager.cc:2147] Created new table and tablet info
0502 02:02:53.227416 (+1322421us) catalog_manager.cc:2173] Created new table in HMS catalog
0502 02:02:53.239091 (+ 11675us) catalog_manager.cc:2207] Wrote table and tablets to system table
0502 02:02:53.240095 (+ 1004us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0502 02:02:53.240607 (+ 512us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0502 02:02:53.229042 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 02:02:53.229262 (+ 220us) write_op.cc:432] Acquiring schema lock in shared mode
0502 02:02:53.229270 (+ 8us) write_op.cc:435] Acquired schema lock
0502 02:02:53.229273 (+ 3us) tablet.cc:662] Decoding operations
0502 02:02:53.229420 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0502 02:02:53.229438 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0502 02:02:53.229445 (+ 7us) tablet.cc:685] Acquiring locks for 2 operations
0502 02:02:53.229551 (+ 106us) tablet.cc:701] Row locks acquired
0502 02:02:53.229553 (+ 2us) write_op.cc:260] PREPARE: finished
0502 02:02:53.229603 (+ 50us) write_op.cc:270] Start()
0502 02:02:53.229645 (+ 42us) write_op.cc:276] Timestamp: P: 1777687373229564 usec, L: 0
0502 02:02:53.229649 (+ 4us) op_driver.cc:348] REPLICATION: starting
0502 02:02:53.229922 (+ 273us) log.cc:844] Serialized 432 byte log entry
0502 02:02:53.235684 (+ 5762us) op_driver.cc:464] REPLICATION: finished
0502 02:02:53.237241 (+ 1557us) write_op.cc:301] APPLY: starting
0502 02:02:53.237455 (+ 214us) tablet.cc:1367] starting BulkCheckPresence
0502 02:02:53.237725 (+ 270us) tablet.cc:1370] finished BulkCheckPresence
0502 02:02:53.237771 (+ 46us) tablet.cc:1372] starting ApplyRowOperation cycle
0502 02:02:53.238260 (+ 489us) tablet.cc:1383] finished ApplyRowOperation cycle
0502 02:02:53.238320 (+ 60us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 02:02:53.238328 (+ 8us) write_op.cc:312] APPLY: finished
0502 02:02:53.238679 (+ 351us) log.cc:844] Serialized 28 byte log entry
0502 02:02:53.238850 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0502 02:02:53.238878 (+ 28us) write_op.cc:454] Released schema lock
0502 02:02:53.238888 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0502 02:02:53.238993 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0502 02:02:53.238996 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":7120,"Hive Metastore.run_cpu_time_us":1110,"Hive Metastore.run_wall_time_us":1306590,"child_traces":[["op",{"apply.queue_time_us":1459,"apply.run_cpu_time_us":1751,"apply.run_wall_time_us":1752,"num_ops":2,"prepare.queue_time_us":166,"prepare.run_cpu_time_us":1087,"prepare.run_wall_time_us":1087,"raft.queue_time_us":1347,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":324,"replication_time_us":5987,"thread_start_us":1775,"threads_started":1}]]}
W20260502 02:02:53.255366 3184 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a61cfc1553f64513ad3755c97eebe4b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 02:02:53.256189 3706 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.168.253:44041)
W20260502 02:02:53.255328 3183 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1377e577a29b415a8c3c80e6b9d577a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260502 02:02:53.386978 3184 catalog_manager.cc:2174] created HMS catalog entry for table [id=59a63f560e594258bff922b5750b367b]
I20260502 02:02:53.394574 3184 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53004 (request call id 2) took 1516 ms. Trace:
I20260502 02:02:53.394755 3184 rpcz_store.cc:276] 0502 02:02:51.878171 (+ 0us) service_pool.cc:167] Inserting onto call queue
0502 02:02:51.878937 (+ 766us) service_pool.cc:224] Handling call
0502 02:02:53.255689 (+1376752us) catalog_manager.cc:2101] Acquired catalog manager lock
0502 02:02:53.256350 (+ 661us) catalog_manager.cc:2147] Created new table and tablet info
0502 02:02:53.386892 (+130542us) catalog_manager.cc:2173] Created new table in HMS catalog
0502 02:02:53.394034 (+ 7142us) catalog_manager.cc:2207] Wrote table and tablets to system table
0502 02:02:53.394300 (+ 266us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0502 02:02:53.394515 (+ 215us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0502 02:02:53.388050 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 02:02:53.388209 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0502 02:02:53.388217 (+ 8us) write_op.cc:435] Acquired schema lock
0502 02:02:53.388219 (+ 2us) tablet.cc:662] Decoding operations
0502 02:02:53.388322 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0502 02:02:53.388340 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0502 02:02:53.388346 (+ 6us) tablet.cc:685] Acquiring locks for 2 operations
0502 02:02:53.388435 (+ 89us) tablet.cc:701] Row locks acquired
0502 02:02:53.388438 (+ 3us) write_op.cc:260] PREPARE: finished
0502 02:02:53.388485 (+ 47us) write_op.cc:270] Start()
0502 02:02:53.388526 (+ 41us) write_op.cc:276] Timestamp: P: 1777687373388432 usec, L: 0
0502 02:02:53.388529 (+ 3us) op_driver.cc:348] REPLICATION: starting
0502 02:02:53.388750 (+ 221us) log.cc:844] Serialized 432 byte log entry
0502 02:02:53.392448 (+ 3698us) op_driver.cc:464] REPLICATION: finished
0502 02:02:53.392595 (+ 147us) write_op.cc:301] APPLY: starting
0502 02:02:53.392614 (+ 19us) tablet.cc:1367] starting BulkCheckPresence
0502 02:02:53.392666 (+ 52us) tablet.cc:1370] finished BulkCheckPresence
0502 02:02:53.392702 (+ 36us) tablet.cc:1372] starting ApplyRowOperation cycle
0502 02:02:53.392851 (+ 149us) tablet.cc:1383] finished ApplyRowOperation cycle
0502 02:02:53.392891 (+ 40us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 02:02:53.392899 (+ 8us) write_op.cc:312] APPLY: finished
0502 02:02:53.392995 (+ 96us) log.cc:844] Serialized 28 byte log entry
0502 02:02:53.393083 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0502 02:02:53.393114 (+ 31us) write_op.cc:454] Released schema lock
0502 02:02:53.393122 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0502 02:02:53.393209 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0502 02:02:53.393213 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1407,"Hive Metastore.run_cpu_time_us":427,"Hive Metastore.run_wall_time_us":128675,"spinlock_wait_cycles":27264,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":602,"apply.run_wall_time_us":602,"num_ops":2,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":868,"prepare.run_wall_time_us":868,"raft.queue_time_us":505,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":256,"replication_time_us":3873}]]}
W20260502 02:02:53.455142 3706 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.168.253:44041)
I20260502 02:02:53.480360 3183 catalog_manager.cc:2174] created HMS catalog entry for table [id=3961b3d648aa43529e1e065696f47b35]
I20260502 02:02:53.490876 3183 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53004 (request call id 3) took 1604 ms. Trace:
I20260502 02:02:53.491055 3183 rpcz_store.cc:276] 0502 02:02:51.886051 (+ 0us) service_pool.cc:167] Inserting onto call queue
0502 02:02:51.886531 (+ 480us) service_pool.cc:224] Handling call
0502 02:02:53.256801 (+1370270us) catalog_manager.cc:2101] Acquired catalog manager lock
0502 02:02:53.257596 (+ 795us) catalog_manager.cc:2147] Created new table and tablet info
0502 02:02:53.480192 (+222596us) catalog_manager.cc:2173] Created new table in HMS catalog
0502 02:02:53.490512 (+ 10320us) catalog_manager.cc:2207] Wrote table and tablets to system table
0502 02:02:53.490695 (+ 183us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0502 02:02:53.490818 (+ 123us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0502 02:02:53.481620 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 02:02:53.481760 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0502 02:02:53.481766 (+ 6us) write_op.cc:435] Acquired schema lock
0502 02:02:53.481768 (+ 2us) tablet.cc:662] Decoding operations
0502 02:02:53.481852 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0502 02:02:53.481867 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0502 02:02:53.481871 (+ 4us) tablet.cc:685] Acquiring locks for 2 operations
0502 02:02:53.481955 (+ 84us) tablet.cc:701] Row locks acquired
0502 02:02:53.481956 (+ 1us) write_op.cc:260] PREPARE: finished
0502 02:02:53.481997 (+ 41us) write_op.cc:270] Start()
0502 02:02:53.482030 (+ 33us) write_op.cc:276] Timestamp: P: 1777687373481937 usec, L: 0
0502 02:02:53.482033 (+ 3us) op_driver.cc:348] REPLICATION: starting
0502 02:02:53.482309 (+ 276us) log.cc:844] Serialized 432 byte log entry
0502 02:02:53.488499 (+ 6190us) op_driver.cc:464] REPLICATION: finished
0502 02:02:53.488862 (+ 363us) write_op.cc:301] APPLY: starting
0502 02:02:53.488891 (+ 29us) tablet.cc:1367] starting BulkCheckPresence
0502 02:02:53.488951 (+ 60us) tablet.cc:1370] finished BulkCheckPresence
0502 02:02:53.488986 (+ 35us) tablet.cc:1372] starting ApplyRowOperation cycle
0502 02:02:53.489111 (+ 125us) tablet.cc:1383] finished ApplyRowOperation cycle
0502 02:02:53.489251 (+ 140us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 02:02:53.489260 (+ 9us) write_op.cc:312] APPLY: finished
0502 02:02:53.489371 (+ 111us) log.cc:844] Serialized 28 byte log entry
0502 02:02:53.489459 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0502 02:02:53.489488 (+ 29us) write_op.cc:454] Released schema lock
0502 02:02:53.489496 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0502 02:02:53.489598 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0502 02:02:53.489602 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":129147,"Hive Metastore.run_cpu_time_us":355,"Hive Metastore.run_wall_time_us":93165,"spinlock_wait_cycles":5504,"child_traces":[["op",{"apply.queue_time_us":264,"apply.run_cpu_time_us":741,"apply.run_wall_time_us":739,"num_ops":2,"prepare.queue_time_us":78,"prepare.run_cpu_time_us":1154,"prepare.run_wall_time_us":1155,"raft.queue_time_us":198,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":411,"replication_time_us":6419}]]}
I20260502 02:02:53.492527 3761 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 02:02:53.497026 3761 catalog_manager.cc:679] Loaded metadata for table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35]
I20260502 02:02:53.502578 3761 catalog_manager.cc:679] Loaded metadata for table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb]
I20260502 02:02:53.504001 3761 catalog_manager.cc:679] Loaded metadata for table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b]
I20260502 02:02:53.506784 3761 tablet_loader.cc:96] loaded metadata for tablet 5472a30702464539a6e121e54ef14158 (table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b])
I20260502 02:02:53.507371 3761 tablet_loader.cc:96] loaded metadata for tablet a42e7434f01e4d729db975687178a448 (table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb])
I20260502 02:02:53.507848 3761 tablet_loader.cc:96] loaded metadata for tablet e46043547ef04e27b5388ca35f511838 (table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35])
I20260502 02:02:53.508466 3761 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 02:02:53.509888 3761 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:53.510070 3761 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 02:02:53.512075 3761 catalog_manager.cc:1514] Loading token signing keys...
I20260502 02:02:53.514210 3761 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Loaded TSK: 0
I20260502 02:02:53.515350 3761 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 02:02:53.518080 3761 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 02:02:53.520557 3761 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 02:02:53.560176 3184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_15a6efb8f407476ba8bcc483a784f4c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:53.561364 3183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_8ddc8addf29b495d8eb6b6e9f95577f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:53.561524 3181 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_11d590196f20472e88078f6174a018fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:53.561604 3176 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_9888762efb7a4f7e86f9bd8f75b8097f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:53.562348 3185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_6020a1239f724c2bb65f4e894b46745b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:53.562598 3177 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53004:
table { table_name: "default.Table_a61cfc1553f64513ad3755c97eebe4b0" } new_table_name: "default.Table_29554c7276624001867e6e589b54c6b0" modify_external_catalogs: true
I20260502 02:02:53.562909 3182 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:53.563320 3178 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53004:
table { table_name: "default.Table_90436a402bba47a3a912ef82a27459b1" } new_table_name: "default.Table_023e1cf0843b406d905f1b100e958a74" modify_external_catalogs: true
I20260502 02:02:53.563668 3180 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_e82eadcb627540889fd970fec3f1bb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:53.564018 3179 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53004:
name: "default.Table_b795fd960741406aab13f8b310fe8812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 02:02:53.585027 3240 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_6020a1239f724c2bb65f4e894b46745b, message={"server":"","servicePrincipal":"","db":"default","table":"table_6020a1239f724c2bb65f4e894b46745b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6020a1239f724c2bb65f4e894b46745b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_6020a1239f724c2bb65f4e894b46745b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6020a1239f724c2bb65f4e894b46745b\",\"kudu.table_id\":\"69699a1909664b5bae9ab3ddc66ad823\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_90436a402bba47a3a912ef82a27459b1, message={"server":"","servicePrincipal":"","db":"default","table":"table_90436a402bba47a3a912ef82a27459b1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_90436a402bba47a3a912ef82a27459b1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_90436a402bba47a3a912ef82a27459b1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_90436a402bba47a3a912ef82a27459b1\",\"kudu.table_id\":\"4d8c0f498fbf4a849444ecf94fa8f3eb\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fc5df210dcd google::LogMessage::Fail() at ??:0
@ 0x7fc5df214b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc5df2107cc google::LogMessage::Flush() at ??:0
@ 0x7fc5df210a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fc5f5ce9574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fc5f5ce7e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fc5f5cefa9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fc5f5cef919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fc5e1fa9529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc5e55976db start_thread at ??:0
@ 0x7fc5de0c161f clone at ??:0
W20260502 02:02:53.667196 3765 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:53.667718 3765 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:53.667903 3765 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:53.686487 3765 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:02:53.686604 3765 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:53.686681 3765 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:02:53.686733 3765 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:02:53.714433 3765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.168.253
--webserver_port=33809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:53.717226 3765 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:53.721529 3765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:53.734584 3796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:53.734714 3795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:53.735890 3798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:53.738317 3765 server_base.cc:1061] running on GCE node
I20260502 02:02:53.739816 3765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:53.743484 3765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:53.745230 3765 hybrid_clock.cc:648] HybridClock initialized: now 1777687373745174 us; error 86 us; skew 500 ppm
I20260502 02:02:53.745760 3765 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:53.749620 3765 webserver.cc:492] Webserver started at http://127.2.168.253:33809/ using document root <none> and password file <none>
I20260502 02:02:53.750633 3765 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:53.750828 3765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:53.759167 3765 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260502 02:02:53.763690 3804 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:53.765921 3765 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260502 02:02:53.766187 3765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:02:53.767053 3765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:53.798578 3765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:53.799760 3765 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:53.800139 3765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:53.832273 3765 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.253:44041
I20260502 02:02:53.832188 3855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.253:44041 every 8 connection(s)
I20260502 02:02:53.834818 3765 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
I20260502 02:02:53.843315 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3765
I20260502 02:02:53.848271 3856 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:02:53.855649 3856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap starting.
I20260502 02:02:53.873209 3856 log.cc:826] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Log is configured to *not* fsync() on all Append() calls
I20260502 02:02:53.887099 3560 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:02:53.896625 3856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 02:02:53.897850 3856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap complete.
I20260502 02:02:53.903350 3698 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:02:53.907115 3856 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:53.909019 3856 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6641b5408d2947bdb1a7d508a61f17e0, State: Initialized, Role: FOLLOWER
I20260502 02:02:53.911006 3856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:53.913933 3864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:53.914824 3864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:53.915593 3856 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:02:53.960767 3878 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:53.960935 3878 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: loading cluster ID for follower catalog manager: success
I20260502 02:02:53.966073 3878 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: acquiring CA information for follower catalog manager: success
I20260502 02:02:53.969662 3878 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:02:54.004184 3883 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32ce575a90d74bd588ce4dd4930dc6ef)
I20260502 02:02:54.006168 3883 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:54.008693 3883 leader_election.cc:290] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:02:54.034003 3708 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:02:54.033999 3586 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:02:54.035003 3310 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:02:54.035961 3030 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:02:54.036741 3030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 9 similar messages]
W20260502 02:02:54.049150 3884 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 02:02:54.001090 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.168.252:41437 (local address 127.0.0.1:53008)
0502 02:02:54.001939 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0502 02:02:54.001961 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0502 02:02:54.003906 (+ 1945us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0502 02:02:54.047159 (+ 43253us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":403,"threads_started":1}
I20260502 02:02:54.049886 3765 master_runner.cc:186] Error getting master registration for 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20260502 02:02:54.060808 3765 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 02:02:54.069131 3030 leader_election.cc:336] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:54.079677 3831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:02:54.080470 3831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f40c77f3a794ac1b717b2a4400f34e7 in term 2.
I20260502 02:02:54.082195 3028 leader_election.cc:304] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0, 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef
I20260502 02:02:54.082674 3883 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260502 02:02:54.082806 3883 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 32ce575a90d74bd588ce4dd4930dc6ef)
I20260502 02:02:54.082918 3883 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 3 FOLLOWER]: Advancing to term 4
I20260502 02:02:54.083081 3422 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:02:54.083348 3422 heartbeater.cc:499] Master 127.2.168.253:44041 was elected leader, sending a full tablet report...
I20260502 02:02:54.085951 3821 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:58363; Asking this server to re-register.
I20260502 02:02:54.086931 3422 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:54.086647 3883 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:54.087894 3883 leader_election.cc:290] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 4 election: Requested vote from peers 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:54.088627 3831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0"
I20260502 02:02:54.088891 3831 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 2 FOLLOWER]: Advancing to term 4
I20260502 02:02:54.089010 3821 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
W20260502 02:02:54.091673 3030 leader_election.cc:336] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:54.095815 3831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f40c77f3a794ac1b717b2a4400f34e7 in term 4.
I20260502 02:02:54.097618 3028 leader_election.cc:304] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0, 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef
I20260502 02:02:54.098129 3883 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260502 02:02:54.098814 3883 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 LEADER]: Becoming Leader. State: Replica: 7f40c77f3a794ac1b717b2a4400f34e7, State: Running, Role: LEADER
I20260502 02:02:54.099391 3883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:54.102028 3890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f40c77f3a794ac1b717b2a4400f34e7. Latest consensus state: current_term: 4 leader_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:54.102336 3890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:54.103319 3891 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 02:02:54.107120 3891 catalog_manager.cc:679] Loaded metadata for table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35]
I20260502 02:02:54.107875 3891 catalog_manager.cc:679] Loaded metadata for table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb]
I20260502 02:02:54.109053 3891 catalog_manager.cc:679] Loaded metadata for table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b]
I20260502 02:02:54.111472 3891 tablet_loader.cc:96] loaded metadata for tablet 5472a30702464539a6e121e54ef14158 (table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b])
I20260502 02:02:54.111878 3891 tablet_loader.cc:96] loaded metadata for tablet a42e7434f01e4d729db975687178a448 (table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb])
I20260502 02:02:54.112216 3891 tablet_loader.cc:96] loaded metadata for tablet e46043547ef04e27b5388ca35f511838 (table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35])
I20260502 02:02:54.112543 3891 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 02:02:54.113919 3891 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:54.114032 3891 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 02:02:54.115787 3891 catalog_manager.cc:1514] Loading token signing keys...
I20260502 02:02:54.117100 3891 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Loaded TSK: 0
I20260502 02:02:54.120708 3891 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 02:02:54.122866 3891 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 02:02:54.125000 3891 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 02:02:54.139670 3043 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59918:
name: "default.Table_96139610dbdf44ab8a2859b996c43cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:54.141556 3037 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59918:
name: "default.Table_15a6efb8f407476ba8bcc483a784f4c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:54.140350 3041 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59918:
name: "default.Table_9888762efb7a4f7e86f9bd8f75b8097f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:54.141732 3036 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59918:
name: "default.Table_8ddc8addf29b495d8eb6b6e9f95577f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:54.140522 3040 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59918:
table { table_name: "default.Table_a61cfc1553f64513ad3755c97eebe4b0" } new_table_name: "default.Table_29554c7276624001867e6e589b54c6b0" modify_external_catalogs: true
I20260502 02:02:54.141067 3038 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59918:
name: "default.Table_b795fd960741406aab13f8b310fe8812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:54.140946 3044 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59918:
name: "default.Table_68c27c450351493fb97b580bd1afc54e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:54.140655 3039 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59918:
table { table_name: "default.Table_90436a402bba47a3a912ef82a27459b1" } new_table_name: "default.Table_023e1cf0843b406d905f1b100e958a74" modify_external_catalogs: true
I20260502 02:02:54.140100 3042 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59918:
table { table_name: "default.Table_1377e577a29b415a8c3c80e6b9d577a3" } new_table_name: "default.Table_3b6d9792a7474737a864c6a076fa9ba7" modify_external_catalogs: true
I20260502 02:02:54.142249 3035 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59918:
name: "default.Table_6020a1239f724c2bb65f4e894b46745b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 02:02:54.153990 3289 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_6020a1239f724c2bb65f4e894b46745b, message={"server":"","servicePrincipal":"","db":"default","table":"table_6020a1239f724c2bb65f4e894b46745b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6020a1239f724c2bb65f4e894b46745b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_6020a1239f724c2bb65f4e894b46745b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6020a1239f724c2bb65f4e894b46745b\",\"kudu.table_id\":\"69699a1909664b5bae9ab3ddc66ad823\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_90436a402bba47a3a912ef82a27459b1, message={"server":"","servicePrincipal":"","db":"default","table":"table_90436a402bba47a3a912ef82a27459b1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_90436a402bba47a3a912ef82a27459b1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_90436a402bba47a3a912ef82a27459b1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_90436a402bba47a3a912ef82a27459b1\",\"kudu.table_id\":\"4d8c0f498fbf4a849444ecf94fa8f3eb\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fd82fe85dcd google::LogMessage::Fail() at ??:0
@ 0x7fd82fe89b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd82fe857cc google::LogMessage::Flush() at ??:0
@ 0x7fd82fe85a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd84695e574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fd84695ce21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fd846964a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fd846964919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fd832c1e529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd83620c6db start_thread at ??:0
@ 0x7fd82ed3661f clone at ??:0
I20260502 02:02:54.394053 3820 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:35815; Asking this server to re-register.
I20260502 02:02:54.395128 3560 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:54.397518 3820 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:02:54.407094 3820 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:55343; Asking this server to re-register.
I20260502 02:02:54.407977 3698 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:54.409718 3820 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
W20260502 02:02:54.508430 3586 connection.cc:570] client connection to 127.2.168.254:43423 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:02:54.509383 3310 connection.cc:570] client connection to 127.2.168.254:43423 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:02:54.510092 3708 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.168.254:43423)
W20260502 02:02:54.511118 3708 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.168.254:43423)
I20260502 02:02:54.640923 3905 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:02:54.641141 3905 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:54.643280 3905 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:02:54.645301 3807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:02:54.647008 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:02:54.647387 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:54.647518 3807 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:02:54.648000 3905 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260502 02:02:55.232169 3909 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:02:55.232410 3909 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:55.233502 3909 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:02:55.236919 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:02:55.237460 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:55.237697 3807 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:02:55.238061 3909 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260502 02:02:56.268050 3913 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:02:56.268255 3913 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:56.269071 3913 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:02:56.272140 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:02:56.272502 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:56.272621 3807 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:02:56.272975 3913 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260502 02:02:57.051542 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.254:43423
--webserver_interface=127.2.168.254
--webserver_port=44147
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 02:02:57.584273 3917 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:02:57.584661 3917 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:02:57.584903 3917 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:02:57.594942 3917 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:02:57.595067 3917 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:02:57.595180 3917 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:02:57.595266 3917 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:02:57.607398 3917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.254:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.168.254
--webserver_port=44147
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:02:57.610219 3917 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:02:57.612485 3917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:02:57.623303 3925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:57.623522 3923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:02:57.624251 3922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:02:57.624451 3917 server_base.cc:1061] running on GCE node
I20260502 02:02:57.625629 3917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:02:57.627486 3917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:02:57.628764 3917 hybrid_clock.cc:648] HybridClock initialized: now 1777687377628721 us; error 83 us; skew 500 ppm
I20260502 02:02:57.629289 3917 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:02:57.632122 3917 webserver.cc:492] Webserver started at http://127.2.168.254:44147/ using document root <none> and password file <none>
I20260502 02:02:57.633071 3917 fs_manager.cc:362] Metadata directory not provided
I20260502 02:02:57.633201 3917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:02:57.639904 3917 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260502 02:02:57.643952 3931 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:02:57.646847 3917 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260502 02:02:57.647087 3917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
uuid: "7f40c77f3a794ac1b717b2a4400f34e7"
format_stamp: "Formatted at 2026-05-02 02:02:47 on dist-test-slave-n6ms"
I20260502 02:02:57.648022 3917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:02:57.674489 3917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:02:57.675767 3917 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:02:57.676083 3917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:02:57.702236 3917 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.254:43423
I20260502 02:02:57.702370 3982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.254:43423 every 8 connection(s)
I20260502 02:02:57.704106 3917 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-0/data/info.pb
I20260502 02:02:57.705410 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 3917
I20260502 02:02:57.713994 3983 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:02:57.718901 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Bootstrap starting.
I20260502 02:02:57.730484 3983 log.cc:826] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Log is configured to *not* fsync() on all Append() calls
I20260502 02:02:57.746316 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 02:02:57.747217 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Bootstrap complete.
I20260502 02:02:57.755934 3983 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:57.758011 3983 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f40c77f3a794ac1b717b2a4400f34e7, State: Initialized, Role: FOLLOWER
I20260502 02:02:57.759253 3983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 4.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:57.760300 3987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:57.760715 3987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:02:57.761915 3983 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:02:57.785002 4001 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:57.785609 4001 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: loading cluster ID for follower catalog manager: success
I20260502 02:02:57.794996 4001 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: acquiring CA information for follower catalog manager: success
I20260502 02:02:57.798408 4001 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:02:57.815043 4005 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:02:57.815742 4005 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:57.823812 4005 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:57.832947 3917 master_runner.cc:186] Error getting master registration for 127.2.168.252:41437: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111),
I20260502 02:02:57.840936 3917 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 02:02:57.844269 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:57.844732 3958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6641b5408d2947bdb1a7d508a61f17e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" is_pre_election: true
I20260502 02:02:57.845644 3958 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6641b5408d2947bdb1a7d508a61f17e0 for term 5 because replica has last-logged OpId of term: 4 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260502 02:02:57.846812 3807 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:02:57.847556 4005 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260502 02:02:58.033738 3697 heartbeater.cc:344] Connected to a master server at 127.2.168.254:43423
I20260502 02:02:58.034839 3421 heartbeater.cc:344] Connected to a master server at 127.2.168.254:43423
I20260502 02:02:58.041712 3947 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:54555; Asking this server to re-register.
I20260502 02:02:58.042415 3948 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:51435; Asking this server to re-register.
I20260502 02:02:58.042728 3421 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:58.043571 3697 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:58.045761 3948 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:02:58.045919 3947 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:02:58.118081 3559 heartbeater.cc:344] Connected to a master server at 127.2.168.254:43423
I20260502 02:02:58.120363 3947 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:47013; Asking this server to re-register.
I20260502 02:02:58.121238 3559 heartbeater.cc:461] Registering TS with master...
I20260502 02:02:58.123431 3947 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:02:58.142086 3987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:02:58.142584 3987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:58.145105 3987 leader_election.cc:290] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
I20260502 02:02:58.146008 3831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
W20260502 02:02:58.146029 3934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:58.146314 3831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f40c77f3a794ac1b717b2a4400f34e7 in term 4.
I20260502 02:02:58.147264 3932 leader_election.cc:304] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0, 7f40c77f3a794ac1b717b2a4400f34e7; no voters:
I20260502 02:02:58.147949 3987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260502 02:02:58.148149 3987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 02:02:58.148303 3987 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 4 FOLLOWER]: Advancing to term 5
I20260502 02:02:58.157297 3987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
W20260502 02:02:58.159476 3934 leader_election.cc:336] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:58.160552 3831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0"
I20260502 02:02:58.160851 3831 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 4 FOLLOWER]: Advancing to term 5
I20260502 02:02:58.160843 3987 leader_election.cc:290] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 5 election: Requested vote from peers 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:02:58.162281 3934 leader_election.cc:336] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:02:58.163995 3831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f40c77f3a794ac1b717b2a4400f34e7 in term 5.
I20260502 02:02:58.165002 3932 leader_election.cc:304] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0, 7f40c77f3a794ac1b717b2a4400f34e7; no voters: 32ce575a90d74bd588ce4dd4930dc6ef
I20260502 02:02:58.165766 3987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260502 02:02:58.167083 3987 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [term 5 LEADER]: Becoming Leader. State: Replica: 7f40c77f3a794ac1b717b2a4400f34e7, State: Running, Role: LEADER
I20260502 02:02:58.167989 3987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.9, Last appended by leader: 9, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:02:58.172583 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f40c77f3a794ac1b717b2a4400f34e7. Latest consensus state: current_term: 5 leader_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:02:58.173020 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7 [sys.catalog]: This master's current role is: LEADER
I20260502 02:02:58.174053 4014 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 02:02:58.177721 4014 catalog_manager.cc:679] Loaded metadata for table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35]
I20260502 02:02:58.178262 4014 catalog_manager.cc:679] Loaded metadata for table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb]
I20260502 02:02:58.178699 4014 catalog_manager.cc:679] Loaded metadata for table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b]
I20260502 02:02:58.180684 4014 tablet_loader.cc:96] loaded metadata for tablet 5472a30702464539a6e121e54ef14158 (table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b])
I20260502 02:02:58.181061 4014 tablet_loader.cc:96] loaded metadata for tablet a42e7434f01e4d729db975687178a448 (table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb])
I20260502 02:02:58.181387 4014 tablet_loader.cc:96] loaded metadata for tablet e46043547ef04e27b5388ca35f511838 (table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35])
I20260502 02:02:58.181674 4014 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 02:02:58.182714 4014 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:02:58.182801 4014 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 02:02:58.184149 4014 catalog_manager.cc:1514] Loading token signing keys...
I20260502 02:02:58.185271 4014 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 7f40c77f3a794ac1b717b2a4400f34e7: Loaded TSK: 0
I20260502 02:02:58.186933 4014 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 02:02:58.188432 4014 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 02:02:58.189582 4014 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 02:02:58.252341 3948 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_11d590196f20472e88078f6174a018fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.252655 3945 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_6020a1239f724c2bb65f4e894b46745b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.252627 3946 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.257042 3947 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_e82eadcb627540889fd970fec3f1bb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.254874 3943 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_15a6efb8f407476ba8bcc483a784f4c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.256650 3939 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34204:
table { table_name: "default.Table_1377e577a29b415a8c3c80e6b9d577a3" } new_table_name: "default.Table_3b6d9792a7474737a864c6a076fa9ba7" modify_external_catalogs: true
I20260502 02:02:58.258061 3942 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_b795fd960741406aab13f8b310fe8812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.252427 3944 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_8ddc8addf29b495d8eb6b6e9f95577f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.256036 3940 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_68c27c450351493fb97b580bd1afc54e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:02:58.255580 3941 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "default.Table_96139610dbdf44ab8a2859b996c43cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 02:02:58.262390 4000 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_6020a1239f724c2bb65f4e894b46745b, message={"server":"","servicePrincipal":"","db":"default","table":"table_6020a1239f724c2bb65f4e894b46745b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6020a1239f724c2bb65f4e894b46745b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_6020a1239f724c2bb65f4e894b46745b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6020a1239f724c2bb65f4e894b46745b\",\"kudu.table_id\":\"69699a1909664b5bae9ab3ddc66ad823\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_90436a402bba47a3a912ef82a27459b1, message={"server":"","servicePrincipal":"","db":"default","table":"table_90436a402bba47a3a912ef82a27459b1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_90436a402bba47a3a912ef82a27459b1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_90436a402bba47a3a912ef82a27459b1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_90436a402bba47a3a912ef82a27459b1\",\"kudu.table_id\":\"4d8c0f498fbf4a849444ecf94fa8f3eb\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fd6ddcdfdcd google::LogMessage::Fail() at ??:0
@ 0x7fd6ddce3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd6ddcdf7cc google::LogMessage::Flush() at ??:0
@ 0x7fd6ddcdfa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd6f47b8574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fd6f47b6e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fd6f47bea9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fd6f47be919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fd6e0a78529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd6e40666db start_thread at ??:0
@ 0x7fd6dcb9061f clone at ??:0
W20260502 02:02:58.531229 3708 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.168.254:43423)
W20260502 02:02:58.531666 3708 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.168.254:43423)
I20260502 02:02:59.070453 3765 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:02:59.070676 3765 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:00.227458 4028 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:00.227730 4028 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:00.228547 4028 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:00.230294 3807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260502 02:03:00.232076 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:00.232321 3807 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:00.232432 3807 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:00.232753 4028 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260502 02:03:00.774870 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.252:41437
--webserver_interface=127.2.168.252
--webserver_port=38947
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 02:03:01.186403 4033 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:03:01.186739 4033 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:03:01.186890 4033 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:03:01.197307 4033 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:03:01.197481 4033 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:03:01.197572 4033 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:03:01.197705 4033 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:03:01.211800 4033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.252:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.168.252
--webserver_port=38947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:03:01.213753 4033 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:03:01.216178 4033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:03:01.227557 4044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:01.227692 4042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:01.227973 4041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:03:01.229336 4033 server_base.cc:1061] running on GCE node
I20260502 02:03:01.230299 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:03:01.232518 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:03:01.233862 4033 hybrid_clock.cc:648] HybridClock initialized: now 1777687381233777 us; error 88 us; skew 500 ppm
I20260502 02:03:01.234333 4033 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:03:01.236936 4033 webserver.cc:492] Webserver started at http://127.2.168.252:38947/ using document root <none> and password file <none>
I20260502 02:03:01.237773 4033 fs_manager.cc:362] Metadata directory not provided
I20260502 02:03:01.237892 4033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:03:01.245163 4033 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260502 02:03:01.250173 4050 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:03:01.252693 4033 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260502 02:03:01.252993 4033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
uuid: "32ce575a90d74bd588ce4dd4930dc6ef"
format_stamp: "Formatted at 2026-05-02 02:02:49 on dist-test-slave-n6ms"
I20260502 02:03:01.253842 4033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:03:01.285058 4033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:03:01.286615 4033 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:03:01.287017 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:03:01.315488 4033 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.252:41437
I20260502 02:03:01.315372 4101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.252:41437 every 8 connection(s)
I20260502 02:03:01.318253 4033 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
I20260502 02:03:01.320510 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 4033
I20260502 02:03:01.334638 4102 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:03:01.340142 4102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Bootstrap starting.
I20260502 02:03:01.356545 4102 log.cc:826] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Log is configured to *not* fsync() on all Append() calls
I20260502 02:03:01.370688 4102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260502 02:03:01.371322 4102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Bootstrap complete.
I20260502 02:03:01.377839 4102 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:01.378826 4102 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Initialized, Role: FOLLOWER
I20260502 02:03:01.379571 4102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:01.380551 4107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:01.380893 4107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:03:01.381912 4102 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:03:01.397214 4121 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:03:01.397845 4121 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: loading cluster ID for follower catalog manager: success
I20260502 02:03:01.403981 4121 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: acquiring CA information for follower catalog manager: success
I20260502 02:03:01.408668 4121 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:03:01.409684 4033 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:01.419288 4033 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:01.580479 3558 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:03:01.582298 3696 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:03:01.583895 4067 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:34723; Asking this server to re-register.
I20260502 02:03:01.585956 4067 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:47253; Asking this server to re-register.
I20260502 02:03:01.586674 3696 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:01.586786 3558 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:01.588991 4066 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:03:01.589140 4067 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:03:01.592592 3420 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:03:01.594683 4066 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:54571; Asking this server to re-register.
I20260502 02:03:01.595474 3420 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:01.598889 4066 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:03:01.846606 4107 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:01.847013 4107 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:01.851859 4107 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
W20260502 02:03:01.853724 4053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:01.859253 4053 leader_election.cc:336] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:03:01.870240 3831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:03:01.870666 3831 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32ce575a90d74bd588ce4dd4930dc6ef for earlier term 4. Current term is 5.
I20260502 02:03:01.875231 4051 leader_election.cc:400] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32ce575a90d74bd588ce4dd4930dc6ef for earlier term 4. Current term is 5.
I20260502 02:03:01.875411 4051 leader_election.cc:403] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260502 02:03:01.876194 4107 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 3 FOLLOWER]: Advancing to term 5
I20260502 02:03:01.882004 4107 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32ce575a90d74bd588ce4dd4930dc6ef for earlier term 4. Current term is 5.
I20260502 02:03:02.446455 4135 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:02.446874 4135 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:02.448738 4135 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
I20260502 02:03:02.449788 3831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:03:02.450090 3831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 5.
I20260502 02:03:02.450783 4051 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 6641b5408d2947bdb1a7d508a61f17e0; no voters:
I20260502 02:03:02.451588 4135 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260502 02:03:02.451779 4135 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 02:03:02.451930 4135 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 5 FOLLOWER]: Advancing to term 6
W20260502 02:03:02.452107 4053 leader_election.cc:336] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:03:02.455888 4135 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:02.457324 4135 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 6 election: Requested vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
I20260502 02:03:02.458166 3831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0"
I20260502 02:03:02.458427 3831 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 5 FOLLOWER]: Advancing to term 6
W20260502 02:03:02.459520 4053 leader_election.cc:336] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:03:02.460805 3831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 6.
I20260502 02:03:02.461745 4051 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 6641b5408d2947bdb1a7d508a61f17e0; no voters: 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:02.462518 4135 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Leader election won for term 6
I20260502 02:03:02.463506 4135 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 LEADER]: Becoming Leader. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Running, Role: LEADER
I20260502 02:03:02.464241 4135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:02.468466 4138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce575a90d74bd588ce4dd4930dc6ef. Latest consensus state: current_term: 6 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:02.468781 4138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: LEADER
I20260502 02:03:02.469627 4140 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 02:03:02.472741 4140 catalog_manager.cc:679] Loaded metadata for table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35]
I20260502 02:03:02.473466 4140 catalog_manager.cc:679] Loaded metadata for table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb]
I20260502 02:03:02.473937 4140 catalog_manager.cc:679] Loaded metadata for table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b]
I20260502 02:03:02.475936 4140 tablet_loader.cc:96] loaded metadata for tablet 5472a30702464539a6e121e54ef14158 (table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b])
I20260502 02:03:02.476423 4140 tablet_loader.cc:96] loaded metadata for tablet a42e7434f01e4d729db975687178a448 (table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb])
I20260502 02:03:02.476765 4140 tablet_loader.cc:96] loaded metadata for tablet e46043547ef04e27b5388ca35f511838 (table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35])
I20260502 02:03:02.477123 4140 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 02:03:02.478927 4140 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:03:02.479087 4140 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 02:03:02.480662 4140 catalog_manager.cc:1514] Loading token signing keys...
I20260502 02:03:02.481678 4140 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Loaded TSK: 0
I20260502 02:03:02.483788 4140 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 02:03:02.485927 4140 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 02:03:02.487066 4140 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 02:03:02.512372 4067 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36006:
table { table_name: "default.Table_90436a402bba47a3a912ef82a27459b1" } new_table_name: "default.Table_023e1cf0843b406d905f1b100e958a74" modify_external_catalogs: true
I20260502 02:03:02.512379 4066 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36006:
table { table_name: "default.Table_a61cfc1553f64513ad3755c97eebe4b0" } new_table_name: "default.Table_29554c7276624001867e6e589b54c6b0" modify_external_catalogs: true
I20260502 02:03:02.513347 4065 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36006:
name: "default.Table_9888762efb7a4f7e86f9bd8f75b8097f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:02.515614 4064 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36006:
table { table_name: "default.Table_1377e577a29b415a8c3c80e6b9d577a3" } new_table_name: "default.Table_3b6d9792a7474737a864c6a076fa9ba7" modify_external_catalogs: true
I20260502 02:03:02.514464 4061 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36006:
name: "default.Table_b795fd960741406aab13f8b310fe8812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:02.517647 4060 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36006:
name: "default.Table_11d590196f20472e88078f6174a018fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:02.518090 4062 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36006:
name: "default.Table_96139610dbdf44ab8a2859b996c43cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:02.513279 4063 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36006:
name: "default.Table_68c27c450351493fb97b580bd1afc54e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:02.520789 4058 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36006:
name: "default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:02.521790 4059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36006:
name: "default.Table_e82eadcb627540889fd970fec3f1bb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 02:03:02.524658 4120 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_6020a1239f724c2bb65f4e894b46745b, message={"server":"","servicePrincipal":"","db":"default","table":"table_6020a1239f724c2bb65f4e894b46745b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6020a1239f724c2bb65f4e894b46745b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_6020a1239f724c2bb65f4e894b46745b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6020a1239f724c2bb65f4e894b46745b\",\"kudu.table_id\":\"69699a1909664b5bae9ab3ddc66ad823\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_90436a402bba47a3a912ef82a27459b1, message={"server":"","servicePrincipal":"","db":"default","table":"table_90436a402bba47a3a912ef82a27459b1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_90436a402bba47a3a912ef82a27459b1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_90436a402bba47a3a912ef82a27459b1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_90436a402bba47a3a912ef82a27459b1\",\"kudu.table_id\":\"4d8c0f498fbf4a849444ecf94fa8f3eb\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f8f0ae92dcd google::LogMessage::Fail() at ??:0
@ 0x7f8f0ae96b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8f0ae927cc google::LogMessage::Flush() at ??:0
@ 0x7f8f0ae92a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f8f2196b574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f8f21969e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f8f21971a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f8f21971919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7f8f0dc2b529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8f112196db start_thread at ??:0
@ 0x7f8f09d4361f clone at ??:0
W20260502 02:03:02.836591 3586 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:03:02.836665 3310 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:03:02.837313 3448 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:03:02.839850 3708 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.168.252:41437)
W20260502 02:03:02.840610 3708 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.168.252:41437)
I20260502 02:03:03.876277 2723 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task3_zYNP/build/asan/bin/kudu pid 3765
I20260502 02:03:03.885423 3851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260502 02:03:04.026353 2723 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3_zYNP/build/asan/bin/kudu with pid 3765
I20260502 02:03:04.771888 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--webserver_interface=127.2.168.253
--webserver_port=33809
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 02:03:05.236675 4163 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:03:05.237152 4163 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:03:05.237238 4163 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:03:05.246843 4163 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:03:05.246971 4163 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:03:05.247035 4163 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:03:05.247076 4163 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:03:05.261618 4163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.168.253
--webserver_port=33809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:03:05.263520 4163 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:03:05.265743 4163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:03:05.277365 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:05.277491 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:05.278504 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:03:05.280651 4163 server_base.cc:1061] running on GCE node
I20260502 02:03:05.281770 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:03:05.286011 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:03:05.287760 4163 hybrid_clock.cc:648] HybridClock initialized: now 1777687385287692 us; error 51 us; skew 500 ppm
I20260502 02:03:05.288277 4163 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:03:05.291726 4163 webserver.cc:492] Webserver started at http://127.2.168.253:33809/ using document root <none> and password file <none>
I20260502 02:03:05.292992 4163 fs_manager.cc:362] Metadata directory not provided
I20260502 02:03:05.293167 4163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:03:05.300520 4163 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260502 02:03:05.304800 4177 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:03:05.306984 4163 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260502 02:03:05.307301 4163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:03:05.308161 4163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:03:05.366567 4163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:03:05.367564 4163 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:03:05.367858 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:03:05.394064 4163 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.253:44041
I20260502 02:03:05.394093 4228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.253:44041 every 8 connection(s)
I20260502 02:03:05.396198 4163 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
I20260502 02:03:05.403887 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 4163
I20260502 02:03:05.406440 4229 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:03:05.411263 4229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap starting.
I20260502 02:03:05.423527 4229 log.cc:826] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Log is configured to *not* fsync() on all Append() calls
I20260502 02:03:05.440735 4229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 02:03:05.441543 4229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap complete.
I20260502 02:03:05.448161 4229 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:05.449729 4229 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6641b5408d2947bdb1a7d508a61f17e0, State: Initialized, Role: FOLLOWER
I20260502 02:03:05.450634 4229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:05.451658 4233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:05.451992 4233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:03:05.452739 4229 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:03:05.469345 4247 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:03:05.469478 4247 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: loading cluster ID for follower catalog manager: success
I20260502 02:03:05.476922 4247 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: acquiring CA information for follower catalog manager: success
I20260502 02:03:05.483415 4247 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:03:05.481145 4163 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:05.486455 4163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:05.538894 3560 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:05.541177 4194 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:33657; Asking this server to re-register.
I20260502 02:03:05.541997 3560 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:05.543933 4194 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:03:05.562088 3698 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:05.563773 4194 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:57665; Asking this server to re-register.
I20260502 02:03:05.564738 3698 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:05.566392 4194 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:03:05.655398 3422 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:05.657444 4193 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:56757; Asking this server to re-register.
I20260502 02:03:05.658504 3422 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:05.660069 4193 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:03:05.952454 4233 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:05.953058 4233 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:05.956533 4233 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:05.957288 4180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:05.959712 4180 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:05.960017 4180 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:05.960215 4180 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:05.960692 4233 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:06.549311 4253 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:06.549584 4253 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:06.550443 4253 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:06.553890 4180 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:06.554137 4180 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:06.554260 4180 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:06.554590 4253 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:07.554009 4257 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:07.554208 4257 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:07.555161 4257 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:07.558359 4180 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:03:07.558598 4180 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:03:07.558701 4180 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:07.558993 4257 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:07.638126 2723 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task3_zYNP/build/asan/bin/kudu pid 4163
I20260502 02:03:07.653074 4224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260502 02:03:07.775125 2723 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3_zYNP/build/asan/bin/kudu with pid 4163
I20260502 02:03:08.641049 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--webserver_interface=127.2.168.253
--webserver_port=33809
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 02:03:09.109897 4265 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:03:09.110754 4265 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:03:09.110942 4265 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:03:09.121577 4265 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:03:09.121682 4265 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:03:09.121778 4265 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:03:09.121848 4265 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:03:09.135547 4265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.168.253
--webserver_port=33809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:03:09.138023 4265 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:03:09.140522 4265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:03:09.152657 4271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:09.152657 4270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:09.153034 4273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:03:09.154614 4265 server_base.cc:1061] running on GCE node
I20260502 02:03:09.155634 4265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:03:09.158443 4265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:03:09.159874 4265 hybrid_clock.cc:648] HybridClock initialized: now 1777687389159731 us; error 121 us; skew 500 ppm
I20260502 02:03:09.160291 4265 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:03:09.163352 4265 webserver.cc:492] Webserver started at http://127.2.168.253:33809/ using document root <none> and password file <none>
I20260502 02:03:09.164461 4265 fs_manager.cc:362] Metadata directory not provided
I20260502 02:03:09.164693 4265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:03:09.172173 4265 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260502 02:03:09.177023 4279 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:03:09.179212 4265 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260502 02:03:09.179594 4265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:03:09.180580 4265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:03:09.232043 4265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:03:09.233227 4265 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:03:09.233610 4265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:03:09.266060 4265 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.253:44041
I20260502 02:03:09.266080 4330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.253:44041 every 8 connection(s)
I20260502 02:03:09.267988 4265 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
I20260502 02:03:09.275312 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 4265
I20260502 02:03:09.276240 4331 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:03:09.281647 4331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap starting.
I20260502 02:03:09.297267 3422 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:09.299850 4331 log.cc:826] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Log is configured to *not* fsync() on all Append() calls
I20260502 02:03:09.313115 4331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 02:03:09.314087 4331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap complete.
I20260502 02:03:09.320494 4331 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:09.321945 4331 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6641b5408d2947bdb1a7d508a61f17e0, State: Initialized, Role: FOLLOWER
I20260502 02:03:09.322818 4331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:09.323833 4336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:09.324185 4336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:03:09.325197 4331 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:03:09.342347 4265 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:09.352227 4265 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:09.356518 4350 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:03:09.356724 4350 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: loading cluster ID for follower catalog manager: success
I20260502 02:03:09.364209 4350 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: acquiring CA information for follower catalog manager: success
I20260502 02:03:09.368427 4350 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:03:09.571265 3560 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:09.573880 4295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:48789; Asking this server to re-register.
I20260502 02:03:09.574595 3560 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:09.577852 4295 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:03:09.590750 3698 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:09.592651 4295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:51153; Asking this server to re-register.
I20260502 02:03:09.593895 3698 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:09.595513 4295 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:03:09.799541 4336 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:09.799962 4336 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:09.801738 4295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:56273; Asking this server to re-register.
I20260502 02:03:09.803045 3422 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:09.803463 4336 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:09.804214 4282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:03:09.804890 4295 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
W20260502 02:03:09.807775 4282 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:09.808120 4282 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:09.808244 4282 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:09.808750 4336 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:10.340236 4355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:10.340517 4355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:10.341349 4355 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:10.344136 4282 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:10.344372 4282 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:10.344497 4282 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:10.344957 4355 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:11.371665 4359 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:11.371935 4359 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:11.373121 4359 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:11.378242 4282 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:11.378569 4282 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:11.378944 4282 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:11.379321 4359 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:11.425616 2723 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task3_zYNP/build/asan/bin/kudu pid 4265
I20260502 02:03:11.435115 4326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260502 02:03:11.558866 2723 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3_zYNP/build/asan/bin/kudu with pid 4265
I20260502 02:03:12.260396 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--webserver_interface=127.2.168.253
--webserver_port=33809
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 02:03:12.753006 4367 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:03:12.754051 4367 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:03:12.754228 4367 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:03:12.769498 4367 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:03:12.769711 4367 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:03:12.769821 4367 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:03:12.769908 4367 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:03:12.785182 4367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.168.253
--webserver_port=33809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:03:12.790023 4367 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:03:12.795977 4367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:03:12.812227 4372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:12.813143 4375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:12.814716 4373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:03:12.815183 4367 server_base.cc:1061] running on GCE node
I20260502 02:03:12.816599 4367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:03:12.819057 4367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:03:12.820981 4367 hybrid_clock.cc:648] HybridClock initialized: now 1777687392820888 us; error 73 us; skew 500 ppm
I20260502 02:03:12.821935 4367 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:03:12.826130 4367 webserver.cc:492] Webserver started at http://127.2.168.253:33809/ using document root <none> and password file <none>
I20260502 02:03:12.827171 4367 fs_manager.cc:362] Metadata directory not provided
I20260502 02:03:12.827301 4367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:03:12.836786 4367 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.001s sys 0.003s
I20260502 02:03:12.846261 4381 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:03:12.850615 4367 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260502 02:03:12.850919 4367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:03:12.853111 4367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:03:12.887390 4367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:03:12.889109 4367 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:03:12.889554 4367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:03:12.933087 4432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.253:44041 every 8 connection(s)
I20260502 02:03:12.933166 4367 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.253:44041
I20260502 02:03:12.937913 4367 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
I20260502 02:03:12.939532 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 4367
I20260502 02:03:12.955060 4433 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:03:12.962860 4433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap starting.
I20260502 02:03:12.981917 4433 log.cc:826] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Log is configured to *not* fsync() on all Append() calls
I20260502 02:03:12.997323 4433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 02:03:12.998046 4433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap complete.
I20260502 02:03:13.007891 4433 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:13.009130 4433 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6641b5408d2947bdb1a7d508a61f17e0, State: Initialized, Role: FOLLOWER
I20260502 02:03:13.009953 4433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:13.011099 4437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:13.011468 4437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:03:13.013295 4433 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:03:13.039214 4451 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:03:13.039433 4451 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: loading cluster ID for follower catalog manager: success
I20260502 02:03:13.047348 4451 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: acquiring CA information for follower catalog manager: success
I20260502 02:03:13.052244 4451 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:03:13.047266 4367 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:13.055869 4367 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:13.106880 3560 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:13.110869 4398 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:41083; Asking this server to re-register.
I20260502 02:03:13.112231 3560 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:13.114847 4398 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:03:13.120260 3698 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:13.122030 4398 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:52319; Asking this server to re-register.
I20260502 02:03:13.122987 3698 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:13.125509 4398 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:03:13.328411 3422 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:13.330022 4398 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:46841; Asking this server to re-register.
I20260502 02:03:13.330796 3422 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:13.332484 4398 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:03:13.551683 4455 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:13.552074 4455 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:13.554327 4455 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:13.555976 4384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:13.558374 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:13.558883 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:13.559001 4384 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:13.559963 4455 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:14.154647 4458 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:14.154893 4458 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:14.155699 4458 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:14.158767 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:14.159035 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:14.159174 4384 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:14.159584 4458 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:15.005515 4462 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:15.005779 4462 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:15.006546 4462 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:15.010677 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:15.010943 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:15.011054 4384 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:15.011590 4462 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 02:03:15.942800 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.252:41437
--webserver_interface=127.2.168.252
--webserver_port=38947
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 02:03:16.021802 4466 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:16.022014 4466 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:16.022804 4466 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:16.025151 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:16.025542 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:16.025697 4384 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:16.026077 4466 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260502 02:03:16.404415 4465 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:03:16.404922 4465 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:03:16.405074 4465 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:03:16.415444 4465 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:03:16.415632 4465 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:03:16.415764 4465 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:03:16.415850 4465 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:03:16.433802 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.252:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.168.252
--webserver_port=38947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:03:16.435879 4465 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:03:16.439219 4465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:03:16.458436 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:16.459851 4473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:16.459647 4474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:03:16.464062 4465 server_base.cc:1061] running on GCE node
I20260502 02:03:16.466017 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:03:16.468787 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:03:16.470223 4465 hybrid_clock.cc:648] HybridClock initialized: now 1777687396470150 us; error 79 us; skew 500 ppm
I20260502 02:03:16.471259 4465 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:03:16.475521 4465 webserver.cc:492] Webserver started at http://127.2.168.252:38947/ using document root <none> and password file <none>
I20260502 02:03:16.476579 4465 fs_manager.cc:362] Metadata directory not provided
I20260502 02:03:16.476778 4465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:03:16.486078 4465 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20260502 02:03:16.493180 4482 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:03:16.495734 4465 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260502 02:03:16.496011 4465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
uuid: "32ce575a90d74bd588ce4dd4930dc6ef"
format_stamp: "Formatted at 2026-05-02 02:02:49 on dist-test-slave-n6ms"
I20260502 02:03:16.497823 4465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:03:16.524083 4465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:03:16.525172 4465 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:03:16.525523 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:03:16.553341 4533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.252:41437 every 8 connection(s)
I20260502 02:03:16.553303 4465 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.252:41437
I20260502 02:03:16.555529 4465 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-2/data/info.pb
I20260502 02:03:16.561792 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 4465
I20260502 02:03:16.566766 4534 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:03:16.571188 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Bootstrap starting.
I20260502 02:03:16.586125 4534 log.cc:826] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Log is configured to *not* fsync() on all Append() calls
I20260502 02:03:16.605945 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 02:03:16.606818 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Bootstrap complete.
I20260502 02:03:16.616349 4534 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:16.618041 4534 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Initialized, Role: FOLLOWER
I20260502 02:03:16.618854 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:16.620369 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:16.620924 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:03:16.622151 4534 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:03:16.636466 4552 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260502 02:03:16.640182 4465 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:16.647902 4465 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:16.892939 3558 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:03:16.892939 3420 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:03:16.892990 3696 heartbeater.cc:344] Connected to a master server at 127.2.168.252:41437
I20260502 02:03:16.895573 4497 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:52295; Asking this server to re-register.
I20260502 02:03:16.895906 4499 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:53231; Asking this server to re-register.
I20260502 02:03:16.895906 4498 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:41833; Asking this server to re-register.
I20260502 02:03:16.896600 3558 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:16.896657 3420 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:16.896777 3696 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:16.898834 4499 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:03:16.898972 4498 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:03:16.899233 4497 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:03:17.045847 4538 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:17.046119 4538 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:17.048076 4538 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
W20260502 02:03:17.049248 4485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:17.051287 4485 leader_election.cc:336] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:03:17.057886 4408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0" is_pre_election: true
I20260502 02:03:17.058346 4408 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 6.
I20260502 02:03:17.059226 4483 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 6641b5408d2947bdb1a7d508a61f17e0; no voters: 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:17.059959 4538 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260502 02:03:17.060158 4538 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 02:03:17.060322 4538 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 6 FOLLOWER]: Advancing to term 7
I20260502 02:03:17.065907 4538 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:17.066756 4538 leader_election.cc:290] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 7 election: Requested vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 6641b5408d2947bdb1a7d508a61f17e0 (127.2.168.253:44041)
I20260502 02:03:17.067512 4408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "6641b5408d2947bdb1a7d508a61f17e0"
I20260502 02:03:17.067909 4408 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 6 FOLLOWER]: Advancing to term 7
W20260502 02:03:17.069635 4485 leader_election.cc:336] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:03:17.072496 4408 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ce575a90d74bd588ce4dd4930dc6ef in term 7.
I20260502 02:03:17.073186 4483 leader_election.cc:304] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ce575a90d74bd588ce4dd4930dc6ef, 6641b5408d2947bdb1a7d508a61f17e0; no voters: 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:17.073606 4538 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 7 FOLLOWER]: Leader election won for term 7
I20260502 02:03:17.074151 4538 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [term 7 LEADER]: Becoming Leader. State: Replica: 32ce575a90d74bd588ce4dd4930dc6ef, State: Running, Role: LEADER
I20260502 02:03:17.074689 4538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:17.077702 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce575a90d74bd588ce4dd4930dc6ef. Latest consensus state: current_term: 7 leader_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:17.078001 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef [sys.catalog]: This master's current role is: LEADER
I20260502 02:03:17.079067 4562 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 02:03:17.085008 4562 catalog_manager.cc:679] Loaded metadata for table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35]
I20260502 02:03:17.085844 4562 catalog_manager.cc:679] Loaded metadata for table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb]
I20260502 02:03:17.086325 4562 catalog_manager.cc:679] Loaded metadata for table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b]
I20260502 02:03:17.089039 4562 tablet_loader.cc:96] loaded metadata for tablet 5472a30702464539a6e121e54ef14158 (table default.table_a61cfc1553f64513ad3755c97eebe4b0 [id=59a63f560e594258bff922b5750b367b])
I20260502 02:03:17.089459 4562 tablet_loader.cc:96] loaded metadata for tablet a42e7434f01e4d729db975687178a448 (table default.table_90436a402bba47a3a912ef82a27459b1 [id=4d8c0f498fbf4a849444ecf94fa8f3eb])
I20260502 02:03:17.089781 4562 tablet_loader.cc:96] loaded metadata for tablet e46043547ef04e27b5388ca35f511838 (table default.table_1377e577a29b415a8c3c80e6b9d577a3 [id=3961b3d648aa43529e1e065696f47b35])
I20260502 02:03:17.090078 4562 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 02:03:17.091522 4562 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:03:17.091670 4562 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 02:03:17.095204 4562 catalog_manager.cc:1514] Loading token signing keys...
I20260502 02:03:17.097565 4562 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 32ce575a90d74bd588ce4dd4930dc6ef: Loaded TSK: 0
I20260502 02:03:17.099144 4562 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 02:03:17.100291 4562 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 02:03:17.101737 4562 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 02:03:17.269986 4499 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_8ddc8addf29b495d8eb6b6e9f95577f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:17.269991 4497 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_15a6efb8f407476ba8bcc483a784f4c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:17.270447 4498 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_6020a1239f724c2bb65f4e894b46745b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:17.271492 4496 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:17.272360 4495 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_e82eadcb627540889fd970fec3f1bb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:17.273144 4494 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_11d590196f20472e88078f6174a018fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:17.274646 4493 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_b795fd960741406aab13f8b310fe8812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 02:03:17.275074 4492 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42854:
table { table_name: "default.Table_a61cfc1553f64513ad3755c97eebe4b0" } new_table_name: "default.Table_29554c7276624001867e6e589b54c6b0" modify_external_catalogs: true
I20260502 02:03:17.276328 4491 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42854:
table { table_name: "default.Table_90436a402bba47a3a912ef82a27459b1" } new_table_name: "default.Table_023e1cf0843b406d905f1b100e958a74" modify_external_catalogs: true
I20260502 02:03:17.277086 4490 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42854:
name: "default.Table_9888762efb7a4f7e86f9bd8f75b8097f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 02:03:17.279873 4551 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_6020a1239f724c2bb65f4e894b46745b, message={"server":"","servicePrincipal":"","db":"default","table":"table_6020a1239f724c2bb65f4e894b46745b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6020a1239f724c2bb65f4e894b46745b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_6020a1239f724c2bb65f4e894b46745b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6020a1239f724c2bb65f4e894b46745b\",\"kudu.table_id\":\"69699a1909664b5bae9ab3ddc66ad823\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777687372, eventType=CREATE_TABLE, dbName=default, tableName=table_90436a402bba47a3a912ef82a27459b1, message={"server":"","servicePrincipal":"","db":"default","table":"table_90436a402bba47a3a912ef82a27459b1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_90436a402bba47a3a912ef82a27459b1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777687372},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/warehouse/table_90436a402bba47a3a912ef82a27459b1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_90436a402bba47a3a912ef82a27459b1\",\"kudu.table_id\":\"4d8c0f498fbf4a849444ecf94fa8f3eb\",\"transient_lastDdlTime\":\"1777687372\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"1b9a85b5f46c46f897bca1ff6dccb853\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777687372,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f1414c84dcd google::LogMessage::Fail() at ??:0
@ 0x7f1414c88b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1414c847cc google::LogMessage::Flush() at ??:0
@ 0x7f1414c84a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f142b75d574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f142b75be21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f142b763a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f142b763919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7f1417a1d529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f141b00b6db start_thread at ??:0
@ 0x7f1413b3561f clone at ??:0
W20260502 02:03:17.546716 3310 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:03:17.546851 3586 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:03:17.547663 3448 connection.cc:570] client connection to 127.2.168.252:41437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 02:03:17.548512 3708 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.168.252:41437)
W20260502 02:03:17.549013 3708 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.168.252:41437)
I20260502 02:03:18.058297 4367 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:18.058589 4367 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:18.226619 4580 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:18.226856 4580 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:18.227566 4580 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:18.230502 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:18.230798 4384 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:18.230911 4384 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:18.231262 4580 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:18.750630 2723 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task3_zYNP/build/asan/bin/kudu pid 4367
I20260502 02:03:18.765777 4428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260502 02:03:18.900158 2723 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3_zYNP/build/asan/bin/kudu with pid 4367
I20260502 02:03:19.756613 2723 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3_zYNP/build/asan/bin/kudu
/tmp/dist-test-task3_zYNP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--webserver_interface=127.2.168.253
--webserver_port=33809
--builtin_ntp_servers=127.2.168.212:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 02:03:20.275571 4587 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 02:03:20.275949 4587 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 02:03:20.276024 4587 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 02:03:20.285463 4587 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 02:03:20.285584 4587 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 02:03:20.285648 4587 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 02:03:20.285693 4587 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 02:03:20.298353 4587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.168.212:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42181
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.168.253:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.168.253
--webserver_port=33809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 01:43:21 UTC on bdcb31816ec0
build id 11702
ASAN enabled
I20260502 02:03:20.300053 4587 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 02:03:20.301944 4587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 02:03:20.311625 4592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:20.311789 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 02:03:20.311656 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 02:03:20.312888 4587 server_base.cc:1061] running on GCE node
I20260502 02:03:20.313911 4587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 02:03:20.315716 4587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 02:03:20.316975 4587 hybrid_clock.cc:648] HybridClock initialized: now 1777687400316921 us; error 51 us; skew 500 ppm
I20260502 02:03:20.317549 4587 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 02:03:20.319996 4587 webserver.cc:492] Webserver started at http://127.2.168.253:33809/ using document root <none> and password file <none>
I20260502 02:03:20.320948 4587 fs_manager.cc:362] Metadata directory not provided
I20260502 02:03:20.321096 4587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 02:03:20.328923 4587 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260502 02:03:20.332528 4601 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 02:03:20.334528 4587 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260502 02:03:20.334782 4587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data,/tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
uuid: "6641b5408d2947bdb1a7d508a61f17e0"
format_stamp: "Formatted at 2026-05-02 02:02:48 on dist-test-slave-n6ms"
I20260502 02:03:20.335614 4587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 02:03:20.386143 4587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 02:03:20.387171 4587 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 02:03:20.387538 4587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 02:03:20.413616 4587 rpc_server.cc:307] RPC server started. Bound to: 127.2.168.253:44041
I20260502 02:03:20.413834 4652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.168.253:44041 every 8 connection(s)
I20260502 02:03:20.415756 4587 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3_zYNP/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777687344159382-2723-0/minicluster-data/master-1/data/info.pb
I20260502 02:03:20.416271 2723 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3_zYNP/build/asan/bin/kudu as pid 4587
I20260502 02:03:20.425160 4653 sys_catalog.cc:263] Verifying existing consensus state
I20260502 02:03:20.430812 4653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap starting.
I20260502 02:03:20.441358 4653 log.cc:826] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Log is configured to *not* fsync() on all Append() calls
I20260502 02:03:20.453565 4653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 02:03:20.454284 4653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: Bootstrap complete.
I20260502 02:03:20.459908 4653 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:20.461046 4653 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6641b5408d2947bdb1a7d508a61f17e0, State: Initialized, Role: FOLLOWER
I20260502 02:03:20.461828 4653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:20.462771 4657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } } }
I20260502 02:03:20.463049 4657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 02:03:20.463923 4653 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 02:03:20.483932 4672 catalog_manager.cc:1269] Loaded cluster ID: 1b9a85b5f46c46f897bca1ff6dccb853
I20260502 02:03:20.484157 4672 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: loading cluster ID for follower catalog manager: success
I20260502 02:03:20.489642 4672 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: acquiring CA information for follower catalog manager: success
I20260502 02:03:20.492978 4672 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 02:03:20.490835 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:20.496562 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:20.651684 3560 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:20.654091 4618 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6b5c908dbe834186ab31f375cc37931a" instance_seqno: 1777687370997230) as {username='slave'} at 127.2.168.194:37051; Asking this server to re-register.
I20260502 02:03:20.654989 3560 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:20.657020 4618 ts_manager.cc:194] Registered new tserver with Master: 6b5c908dbe834186ab31f375cc37931a (127.2.168.194:38029)
I20260502 02:03:20.663803 3698 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:20.665187 4618 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "579c61089da24972849347b57060a11f" instance_seqno: 1777687371739157) as {username='slave'} at 127.2.168.195:46405; Asking this server to re-register.
I20260502 02:03:20.665808 3698 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:20.667189 4618 ts_manager.cc:194] Registered new tserver with Master: 579c61089da24972849347b57060a11f (127.2.168.195:39731)
I20260502 02:03:20.913235 3422 heartbeater.cc:344] Connected to a master server at 127.2.168.253:44041
I20260502 02:03:20.914842 4618 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "13f4d60cbae24d7681fd5a3800b323d0" instance_seqno: 1777687370398619) as {username='slave'} at 127.2.168.193:32869; Asking this server to re-register.
I20260502 02:03:20.915791 3422 heartbeater.cc:461] Registering TS with master...
I20260502 02:03:20.917295 4618 ts_manager.cc:194] Registered new tserver with Master: 13f4d60cbae24d7681fd5a3800b323d0 (127.2.168.193:33145)
I20260502 02:03:21.055737 4676 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:21.056008 4676 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:21.058297 4676 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:21.059973 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:21.062038 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:21.062314 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:21.062443 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:21.062894 4676 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:21.804575 4680 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:21.804773 4680 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:21.805516 4680 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:21.808060 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:21.808329 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:21.808456 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:21.809016 4680 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:22.662230 4683 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:22.662465 4683 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:22.663354 4683 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:22.666558 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:22.667037 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:22.667162 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:22.667446 4683 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:22.765726 2723 master-stress-test.cc:577] Killing test thread 1/25
I20260502 02:03:23.957099 4686 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:23.957307 4686 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:23.958181 4686 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:23.962633 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:23.963104 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:23.963341 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:23.963788 4686 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:25.498764 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:25.499029 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:25.631412 4691 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:25.631686 4691 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:25.632284 4691 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:25.634796 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:25.635071 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:25.635178 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:25.635500 4691 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:29.303784 4694 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:29.304025 4694 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:29.304946 4694 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:29.306456 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260502 02:03:29.308002 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:29.308208 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:29.308282 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:29.308647 4694 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:30.501016 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:30.501242 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:32.218370 4699 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:32.218647 4699 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:32.219419 4699 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:32.222719 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:32.222949 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:32.223049 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:32.223318 4699 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:35.503306 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:35.503518 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:35.572544 4706 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:35.572821 4706 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:35.573688 4706 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:35.574680 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 02:03:35.576377 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:35.576622 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:35.576771 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:35.577122 4706 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:40.505470 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:40.505767 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:43.842898 4711 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:43.843207 4711 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:43.844326 4711 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:43.845347 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:03:43.849491 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:43.849712 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:43.849805 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:43.850135 4711 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:45.508209 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:45.508484 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:48.737823 4715 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:48.738145 4715 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:48.739193 4715 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:48.742233 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:48.742564 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:48.742867 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:48.743265 4715 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:03:50.510282 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:50.510494 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 02:03:52.659361 3420 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (70 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:03:52.659361 3696 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (70 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:03:52.659755 3558 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (70 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:55.512301 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:03:55.512480 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:03:56.492555 4728 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:03:56.493036 4728 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:03:56.494017 4728 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:03:56.506381 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 02:03:56.508877 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:03:56.509146 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:03:56.509261 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:03:56.509598 4728 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:04:00.514200 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:00.514384 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:05.516578 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:05.516780 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:10.518824 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:10.519054 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:10.585696 4745 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:04:10.585925 4745 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:04:10.586659 4745 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:04:10.587980 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:04:10.590200 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:04:10.590499 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:04:10.590636 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:04:10.591048 4745 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:04:15.521063 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:15.521293 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:20.523240 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:20.523458 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:20.675477 4754 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:04:20.675729 4754 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:04:20.676565 4754 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:04:20.677714 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:04:20.682629 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:04:20.682920 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:04:20.683043 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:04:20.683413 4754 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:04:25.525724 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:25.525987 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:30.531935 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:30.532239 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:35.534637 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:35.534878 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:40.228789 4768 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:04:40.229120 4768 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:04:40.230043 4768 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:04:40.232332 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:04:40.234351 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:04:40.234634 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:04:40.234750 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:04:40.235121 4768 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:04:40.537230 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:40.537439 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:45.040380 4774 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:04:45.040680 4774 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:04:45.041574 4774 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:04:45.044660 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:04:45.045068 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:04:45.045220 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:04:45.045670 4774 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:04:45.539012 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:45.539285 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:04:48.295917 4779 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:04:48.296239 4779 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:04:48.297134 4779 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:04:48.298406 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 02:04:48.300014 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:04:48.300220 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:04:48.300323 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:04:48.300608 4779 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:04:50.541296 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:50.541553 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 02:04:52.850608 3558 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (190 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:04:52.850701 3696 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (190 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:04:52.850986 3420 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (190 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:04:55.543798 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:04:55.544130 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:00.546749 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:00.547096 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:05.549098 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:05.549342 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:07.591447 4796 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:05:07.591737 4796 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:05:07.593014 4796 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:05:07.594782 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:05:07.596937 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:05:07.597143 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:05:07.597278 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:05:07.597584 4796 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:05:10.551195 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:10.551470 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:15.553596 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:15.553902 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:20.555770 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:20.556041 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:20.625799 4607 maintenance_manager.cc:419] P 6641b5408d2947bdb1a7d508a61f17e0: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033379
I20260502 02:05:20.682278 4606 maintenance_manager.cc:643] P 6641b5408d2947bdb1a7d508a61f17e0: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.026s sys 0.004s Metrics: {"bytes_written":6029,"cfile_init":1,"dirs.queue_time_us":558,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":959,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":660,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":3,"thread_start_us":727,"threads_started":2,"wal-append.queue_time_us":505}
I20260502 02:05:20.683835 4607 maintenance_manager.cc:419] P 6641b5408d2947bdb1a7d508a61f17e0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260502 02:05:20.684999 4606 maintenance_manager.cc:643] P 6641b5408d2947bdb1a7d508a61f17e0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20260502 02:05:22.537914 4810 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:05:22.538203 4810 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:05:22.539193 4810 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:05:22.540699 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:05:22.543002 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:05:22.543277 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:05:22.543411 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:05:22.543804 4810 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:05:25.559090 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:25.559638 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:30.561935 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:30.562407 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:32.111429 4843 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:05:32.111716 4843 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:05:32.112526 4843 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:05:32.113799 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:05:32.115558 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:05:32.115751 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
I20260502 02:05:32.115872 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:05:32.116305 4843 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:05:35.564338 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:35.564621 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:38.063457 4863 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:05:38.063824 4863 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:05:38.064831 4863 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:05:38.065832 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:05:38.068271 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:05:38.068491 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:05:38.068770 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:05:38.069192 4863 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:05:40.566751 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:40.567052 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:44.910842 4889 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:05:44.911114 4889 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:05:44.911965 4889 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:05:44.913498 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:05:44.918555 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:05:44.918855 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:05:44.919034 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:05:44.919482 4889 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:05:45.569303 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:45.569613 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:50.572037 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:50.573038 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 02:05:53.051496 3696 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (310 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:05:53.053170 3420 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (310 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:05:53.053508 3558 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (310 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:05:55.575284 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:05:55.575501 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:05:55.947016 4930 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:05:55.947292 4930 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:05:55.948115 4930 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:05:55.949096 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:05:55.951239 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:05:55.951843 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:05:55.951990 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:05:55.952333 4930 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:06:00.578366 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:00.578600 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:02.663441 4953 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:06:02.663895 4953 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:06:02.665714 4953 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:06:02.669565 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:06:02.671515 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:06:02.671728 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:06:02.671808 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:06:02.672075 4953 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:06:05.580787 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:05.581146 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:07.008642 4974 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:06:07.009033 4974 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:06:07.010179 4974 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:06:07.013343 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:06:07.013810 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:06:07.013981 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:06:07.014330 4974 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:06:10.583702 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:10.583925 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:15.586695 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:15.586941 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:20.589648 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:20.589967 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:25.441232 5027 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:06:25.441569 5027 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:06:25.442894 5027 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:06:25.444360 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 02:06:25.446746 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:06:25.446967 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:06:25.447039 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:06:25.447340 5027 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:06:25.591737 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:25.591969 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:30.594496 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:30.594907 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:35.597173 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:35.597407 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:40.600208 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:40.600584 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:42.560153 5080 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:06:42.560474 5080 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:06:42.561506 5080 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:06:42.563576 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:06:42.567602 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:06:42.568192 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:06:42.568310 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:06:42.568892 5080 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:06:45.602942 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:45.603449 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:06:50.605685 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:50.605950 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 02:06:53.265105 3420 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (430 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:06:53.265179 3558 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (430 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:06:53.265516 3696 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (430 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:06:55.608566 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:06:55.608789 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:00.611137 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:00.611447 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:02.436642 5140 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:02.437027 5140 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:02.438045 5140 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:02.440454 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:07:02.442718 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:02.442957 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:02.443035 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:02.443974 5140 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:03.006240 5146 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:03.006513 5146 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:03.007661 5146 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:03.011293 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:03.011629 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:03.011766 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:03.012465 5146 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:05.613924 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:05.614163 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:10.616091 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:10.616369 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:15.618175 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:15.618427 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:18.752972 5192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:18.753209 5192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:18.754288 5192 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:18.755650 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 02:07:18.757268 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:18.757524 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:18.757670 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:18.757951 5192 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:20.620077 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:20.620316 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:25.624158 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:25.624439 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:26.114809 5218 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:26.115639 5218 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:26.116582 5218 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:26.117810 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:07:26.120025 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:26.120260 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:26.120406 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:26.120736 5218 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:27.878754 5224 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:27.879050 5224 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:27.880091 5224 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:27.883380 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:27.883919 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:27.884084 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:27.884912 5224 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:30.626528 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:30.626801 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:35.118611 5246 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:35.118922 5246 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:35.119815 5246 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:35.121439 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 02:07:35.125765 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:35.126049 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:35.126145 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:35.126541 5246 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:35.629019 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:35.629528 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:40.631523 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:40.632007 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:41.771410 5268 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:41.771625 5268 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:41.772454 5268 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:41.773940 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:07:41.775660 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:41.775944 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:41.776055 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:41.776486 5268 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:45.633806 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:45.634160 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 02:07:46.825592 5286 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:46.825925 5286 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:46.826756 5286 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:46.828158 4604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 02:07:46.830446 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:46.830900 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:46.831060 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:46.831557 5286 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:49.765591 4607 maintenance_manager.cc:419] P 6641b5408d2947bdb1a7d508a61f17e0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260502 02:07:49.771153 4606 maintenance_manager.cc:643] P 6641b5408d2947bdb1a7d508a61f17e0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.003s sys 0.002s Metrics: {"dirs.queue_time_us":457,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":121,"thread_start_us":649,"threads_started":2,"wal-append.queue_time_us":453}
I20260502 02:07:50.294138 5298 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 02:07:50.294411 5298 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f40c77f3a794ac1b717b2a4400f34e7" member_type: VOTER last_known_addr { host: "127.2.168.254" port: 43423 } } peers { permanent_uuid: "6641b5408d2947bdb1a7d508a61f17e0" member_type: VOTER last_known_addr { host: "127.2.168.253" port: 44041 } } peers { permanent_uuid: "32ce575a90d74bd588ce4dd4930dc6ef" member_type: VOTER last_known_addr { host: "127.2.168.252" port: 41437 } }
I20260502 02:07:50.295207 5298 leader_election.cc:290] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423), 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437)
W20260502 02:07:50.297919 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f40c77f3a794ac1b717b2a4400f34e7 (127.2.168.254:43423): Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111)
W20260502 02:07:50.298211 4604 leader_election.cc:336] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32ce575a90d74bd588ce4dd4930dc6ef (127.2.168.252:41437): Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
I20260502 02:07:50.298318 4604 leader_election.cc:304] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6641b5408d2947bdb1a7d508a61f17e0; no voters: 32ce575a90d74bd588ce4dd4930dc6ef, 7f40c77f3a794ac1b717b2a4400f34e7
I20260502 02:07:50.298563 5298 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6641b5408d2947bdb1a7d508a61f17e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260502 02:07:50.635582 4587 master_runner.cc:186] Error getting master registration for 127.2.168.254:43423: Network error: Client connection negotiation failed: client connection to 127.2.168.254:43423: connect: Connection refused (error 111),
I20260502 02:07:50.635864 4587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 02:07:53.470211 3696 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (550 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:07:53.470265 3420 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (550 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:07:53.470239 3558 heartbeater.cc:646] Failed to heartbeat to 127.2.168.252:41437 (550 consecutive failures): Network error: Failed to ping master at 127.2.168.252:41437: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:07:54.140600 3706 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
W20260502 02:07:54.140944 3706 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
F20260502 02:07:54.151993 3717 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_96139610dbdf44ab8a2859b996c43cdb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1777687674 (unix time) try "date -d @1777687674" if you are using GNU date ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.157521 3733 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.157521 3733 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160141 3730 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b795fd960741406aab13f8b310fe8812 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160310 *** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.157521 3733 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160141 3730 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b795fd960741406aab13f8b310fe8812 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160310 3725 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11d590196f20472e88078f6174a018fd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.157521 3733 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160141 3730 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b795fd960741406aab13f8b310fe8812 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160310 3725 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11d590196f20472e88078f6174a018fd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160180 3727 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8ddc8addf29b495d8eb6b6e9f95577f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.157521 3733 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160141 3730 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b795fd960741406aab13f8b310fe8812 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160310 3725 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11d590196f20472e88078f6174a018fd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160180 3727 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8ddc8addf29b495d8eb6b6e9f95577f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160900 3732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20260502 02:07:54.151994 3716 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9888762efb7a4f7e86f9bd8f75b8097f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.153491 3726 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71638c7e0a1d4f5e99b156f83cf7d4b7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155041 3731 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.155210 3720 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_68c27c450351493fb97b580bd1afc54e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.156508 3729 master-stress-test.cc:297] Check failed: _s.ok() Bad statusF 20260502 02:07:54.156508 3728 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_15a6efb8f407476ba8bcc483a784f4c5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)Timed out: Error creating table default.Table_e82eadcb627540889fd970fec3f1bb43 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.157521 3733 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160141 3730 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b795fd960741406aab13f8b310fe8812 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160310 3725 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11d590196f20472e88078f6174a018fd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160180 3727 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8ddc8addf29b495d8eb6b6e9f95577f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160900 3732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)F20260502 02:07:54.160907 3721 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6020a1239f724c2bb65f4e894b46745b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.168.254:43423,127.2.168.253:44041,127.2.168.252:41437, num_attempts: 739) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.168.252:41437: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800000aa3) received by PID 2723 (TID 0x7efd0fcb3700) from PID 2723; stack trace: ***
@ 0x7efd22f83980 (unknown) at ??:0
@ 0x7efd1bd73e87 gsignal at ??:0
@ 0x7efd1bd757f1 abort at ??:0
@ 0x7efd1eaaddcd google::LogMessage::Fail() at ??:0
@ 0x7efd1eab1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7efd1eaad7cc google::LogMessage::Flush() at ??:0
@ 0x7efd1eaaef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b582 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7efd1c3fb6df (unknown) at ??:0
@ 0x7efd22f786db start_thread at ??:0
@ 0x7efd1be5661f clone at ??:0