Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-04-30T13:54:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T13:54:19Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskZgxaqK/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskZgxaqK/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/security.properties
2026-04-30 13:54:27: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskZgxaqK/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskZgxaqK/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260430 13:54:36.595204 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:35887
--webserver_interface=127.0.105.62
--webserver_port=0
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260430 13:54:36.949805 706 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:36.950147 706 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:36.950250 706 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:36.959308 706 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:36.959434 706 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:36.959497 706 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:36.959571 706 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:54:36.970932 706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:36.972671 706 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:36.974690 706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:36.984339 712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:36.984340 711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:36.985092 714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:36.985975 706 server_base.cc:1061] running on GCE node
I20260430 13:54:36.986896 706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:36.988727 706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:36.990024 706 hybrid_clock.cc:648] HybridClock initialized: now 1777557276989926 us; error 98 us; skew 500 ppm
I20260430 13:54:36.990424 706 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:36.993186 706 webserver.cc:492] Webserver started at http://127.0.105.62:38453/ using document root <none> and password file <none>
I20260430 13:54:36.994014 706 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:36.994124 706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:36.994489 706 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 13:54:36.997218 706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/instance:
uuid: "d392b5529f4245c397bd78166d0bc001"
format_stamp: "Formatted at 2026-04-30 13:54:36 on dist-test-slave-fmnx"
I20260430 13:54:36.998098 706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal/instance:
uuid: "d392b5529f4245c397bd78166d0bc001"
format_stamp: "Formatted at 2026-04-30 13:54:36 on dist-test-slave-fmnx"
I20260430 13:54:37.004330 706 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260430 13:54:37.008255 720 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:37.009964 706 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260430 13:54:37.010152 706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
uuid: "d392b5529f4245c397bd78166d0bc001"
format_stamp: "Formatted at 2026-04-30 13:54:36 on dist-test-slave-fmnx"
I20260430 13:54:37.010368 706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:37.034849 706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:37.035908 706 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:37.036162 706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:37.058552 706 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.62:35887
I20260430 13:54:37.058568 771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.62:35887 every 8 connection(s)
I20260430 13:54:37.060199 706 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
I20260430 13:54:37.063755 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 706
I20260430 13:54:37.064121 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal/instance
I20260430 13:54:37.064839 772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 13:54:37.066761 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:38801
--webserver_interface=127.0.105.61
--webserver_port=0
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260430 13:54:37.071606 772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } has no permanent_uuid. Determining permanent_uuid...
I20260430 13:54:37.094002 772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } has no permanent_uuid. Determining permanent_uuid...
W20260430 13:54:37.095263 724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:37.097093 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:37.146572 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } attempt: 1
W20260430 13:54:37.148603 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:37.216851 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } attempt: 2
W20260430 13:54:37.218952 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:37.310264 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } attempt: 3
W20260430 13:54:37.312286 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:37.441329 773 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:37.441699 773 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:37.441875 773 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:37.451529 773 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:37.451656 773 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:37.451742 773 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:37.451843 773 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:54:37.455561 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } attempt: 4
W20260430 13:54:37.457404 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:37.464377 773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:37.466125 773 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:37.468047 773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:37.476234 782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:37.476497 781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:37.476822 784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:37.477759 773 server_base.cc:1061] running on GCE node
I20260430 13:54:37.478575 773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:37.480134 773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:37.481350 773 hybrid_clock.cc:648] HybridClock initialized: now 1777557277481299 us; error 54 us; skew 500 ppm
I20260430 13:54:37.481719 773 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:37.484135 773 webserver.cc:492] Webserver started at http://127.0.105.61:43365/ using document root <none> and password file <none>
I20260430 13:54:37.484913 773 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:37.485028 773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:37.485383 773 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 13:54:37.487900 773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/instance:
uuid: "f615c6cf70da4e8f8c4dac485b923dee"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:37.488700 773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal/instance:
uuid: "f615c6cf70da4e8f8c4dac485b923dee"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:37.495033 773 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260430 13:54:37.499143 790 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:37.500842 773 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 13:54:37.501071 773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
uuid: "f615c6cf70da4e8f8c4dac485b923dee"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:37.501291 773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:37.518316 773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:37.519313 773 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:37.519570 773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:37.541678 773 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:38801
I20260430 13:54:37.541697 841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:38801 every 8 connection(s)
I20260430 13:54:37.543210 773 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
I20260430 13:54:37.546679 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 773
I20260430 13:54:37.546911 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal/instance
I20260430 13:54:37.548331 842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 13:54:37.548872 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--webserver_interface=127.0.105.60
--webserver_port=0
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260430 13:54:37.555599 842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } has no permanent_uuid. Determining permanent_uuid...
I20260430 13:54:37.570336 842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } has no permanent_uuid. Determining permanent_uuid...
I20260430 13:54:37.580487 842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
W20260430 13:54:37.581861 792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
W20260430 13:54:37.584134 842 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:37.633477 842 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 1
W20260430 13:54:37.635416 842 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:37.703750 842 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 2
W20260430 13:54:37.705783 842 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:37.756714 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } attempt: 5
I20260430 13:54:37.763897 772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
W20260430 13:54:37.765800 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:37.797097 842 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 3
W20260430 13:54:37.798961 842 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:37.817102 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 1
W20260430 13:54:37.819123 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:37.887398 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 2
W20260430 13:54:37.889572 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
W20260430 13:54:37.920728 843 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:37.921046 843 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:37.921154 843 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:37.929730 843 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:37.929843 843 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:37.929908 843 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:37.929968 843 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:54:37.940804 843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:37.942256 842 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 4
I20260430 13:54:37.942646 843 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260430 13:54:37.944240 842 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:37.944651 843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:37.953420 851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:37.953457 852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:37.956110 843 server_base.cc:1061] running on GCE node
W20260430 13:54:37.957078 854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:37.958046 843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:37.959564 843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:37.960803 843 hybrid_clock.cc:648] HybridClock initialized: now 1777557277960728 us; error 65 us; skew 500 ppm
I20260430 13:54:37.961213 843 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:37.963449 843 webserver.cc:492] Webserver started at http://127.0.105.60:43565/ using document root <none> and password file <none>
I20260430 13:54:37.964258 843 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:37.964365 843 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:37.964702 843 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 13:54:37.967149 843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/instance:
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:37.967963 843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal/instance:
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:37.974186 843 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260430 13:54:37.977880 860 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:37.979382 843 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 13:54:37.979566 843 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:37.979763 843 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:37.995853 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 3
W20260430 13:54:37.997918 772 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:41159: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:38.001801 843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:38.003149 843 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:38.003417 843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:38.024631 911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:41159 every 8 connection(s)
I20260430 13:54:38.024655 843 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:41159
I20260430 13:54:38.026310 843 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
I20260430 13:54:38.027921 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 843
I20260430 13:54:38.028106 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal/instance
I20260430 13:54:38.032506 912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 13:54:38.039258 912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } has no permanent_uuid. Determining permanent_uuid...
I20260430 13:54:38.051893 912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } has no permanent_uuid. Determining permanent_uuid...
I20260430 13:54:38.061828 912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20260430 13:54:38.078701 912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap starting.
I20260430 13:54:38.082772 912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Neither blocks nor log segments found. Creating new log.
I20260430 13:54:38.083992 912 log.cc:826] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:38.087157 912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: No bootstrap required, opened a new log
I20260430 13:54:38.092883 912 raft_consensus.cc:359] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.093250 912 raft_consensus.cc:385] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 13:54:38.093380 912 raft_consensus.cc:740] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Initialized, Role: FOLLOWER
I20260430 13:54:38.094256 912 consensus_queue.cc:260] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.095319 920 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.095564 920 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:38.096539 912 sys_catalog.cc:565] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: configured and running, proceeding with master startup.
W20260430 13:54:38.114585 934 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 13:54:38.114849 934 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 13:54:38.108479 843 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: OK, instance_id { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" instance_seqno: 1777557277052597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260430 13:54:38.118433 843 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:38.175187 772 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 4
I20260430 13:54:38.192561 772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Bootstrap starting.
I20260430 13:54:38.196038 772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Neither blocks nor log segments found. Creating new log.
I20260430 13:54:38.197248 772 log.cc:826] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:38.199928 772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: No bootstrap required, opened a new log
I20260430 13:54:38.206806 772 raft_consensus.cc:359] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.207229 772 raft_consensus.cc:385] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 13:54:38.207363 772 raft_consensus.cc:740] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Initialized, Role: FOLLOWER
I20260430 13:54:38.208079 772 consensus_queue.cc:260] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.209010 939 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.209300 939 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:38.210227 772 sys_catalog.cc:565] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:54:38.223862 920 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:38.224155 920 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.225980 920 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
I20260430 13:54:38.227998 747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001" is_pre_election: true
I20260430 13:54:38.228645 747 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f962b03e872d4cb990156ed62ea6afd3 in term 0.
I20260430 13:54:38.223392 706 master_runner.cc:186] Error getting master registration for 127.0.105.61:38801: OK, instance_id { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" instance_seqno: 1777557277536022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260430 13:54:38.229127 706 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:38.229647 862 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3; no voters:
I20260430 13:54:38.230204 920 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 13:54:38.230337 920 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 13:54:38.230420 920 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 0 FOLLOWER]: Advancing to term 1
W20260430 13:54:38.233870 953 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 13:54:38.233999 953 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 13:54:38.241997 920 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.242888 920 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 1 election: Requested vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
I20260430 13:54:38.243456 842 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } attempt: 5
I20260430 13:54:38.243496 747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001"
I20260430 13:54:38.243726 747 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 0 FOLLOWER]: Advancing to term 1
I20260430 13:54:38.248486 747 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f962b03e872d4cb990156ed62ea6afd3 in term 1.
I20260430 13:54:38.249133 862 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3; no voters:
I20260430 13:54:38.249660 920 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 1 FOLLOWER]: Leader election won for term 1
W20260430 13:54:38.255649 866 tablet.cc:2404] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 13:54:38.257555 842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Bootstrap starting.
I20260430 13:54:38.262410 920 raft_consensus.cc:697] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 1 LEADER]: Becoming Leader. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Running, Role: LEADER
I20260430 13:54:38.263470 920 consensus_queue.cc:237] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.265538 842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Neither blocks nor log segments found. Creating new log.
I20260430 13:54:38.280442 842 log.cc:826] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:38.282357 956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f962b03e872d4cb990156ed62ea6afd3. Latest consensus state: current_term: 1 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.283103 956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: LEADER
W20260430 13:54:38.289213 726 tablet.cc:2404] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 13:54:38.297308 959 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:54:38.299203 842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: No bootstrap required, opened a new log
I20260430 13:54:38.299588 959 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:54:38.309531 842 raft_consensus.cc:359] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.309929 842 raft_consensus.cc:385] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 13:54:38.310050 842 raft_consensus.cc:740] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f615c6cf70da4e8f8c4dac485b923dee, State: Initialized, Role: FOLLOWER
I20260430 13:54:38.310834 842 consensus_queue.cc:260] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:38.312263 961 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.312863 961 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:38.313628 842 sys_catalog.cc:565] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:54:38.313678 816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee"
I20260430 13:54:38.314381 816 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 0 FOLLOWER]: Advancing to term 1
I20260430 13:54:38.313638 817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee" is_pre_election: true
I20260430 13:54:38.322119 747 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Refusing update from remote peer f962b03e872d4cb990156ed62ea6afd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 13:54:38.325822 956 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 13:54:38.335233 816 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f962b03e872d4cb990156ed62ea6afd3 in term 1.
I20260430 13:54:38.336752 817 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Refusing update from remote peer f962b03e872d4cb990156ed62ea6afd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 13:54:38.338985 956 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 13:54:38.354844 939 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f962b03e872d4cb990156ed62ea6afd3. Latest consensus state: current_term: 1 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.355118 939 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:38.358101 960 mvcc.cc:204] Tried to move back new op lower bound from 7280874611988750336 to 7280874611776671744. Current Snapshot: MvccSnapshot[applied={T|T < 7280874611988750336}]
I20260430 13:54:38.358567 956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.358803 956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:38.359089 963 mvcc.cc:204] Tried to move back new op lower bound from 7280874611988750336 to 7280874611776671744. Current Snapshot: MvccSnapshot[applied={T|T < 7280874611988750336}]
I20260430 13:54:38.362442 959 catalog_manager.cc:1357] Generated new cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:38.362567 959 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:54:38.376634 961 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f962b03e872d4cb990156ed62ea6afd3. Latest consensus state: current_term: 1 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.377007 961 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:38.378095 956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.378361 956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:38.382470 939 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.382704 939 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:38.384577 961 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:38.384804 961 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:38.394948 966 mvcc.cc:204] Tried to move back new op lower bound from 7280874611988750336 to 7280874611776671744. Current Snapshot: MvccSnapshot[applied={T|T < 7280874611988750336}]
I20260430 13:54:38.416702 959 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 13:54:38.418493 959 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:54:38.438915 959 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Generated new TSK 0
I20260430 13:54:38.440263 959 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:54:38.441740 959 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:54:38.442926 959 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:54:38.466204 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.105.1:0
--local_ip_for_outbound_sockets=127.0.105.1
--webserver_interface=127.0.105.1
--webserver_port=0
--tserver_master_addrs=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260430 13:54:38.535643 985 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:38.535768 985 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: loading cluster ID for follower catalog manager: success
I20260430 13:54:38.540414 985 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: acquiring CA information for follower catalog manager: success
I20260430 13:54:38.560562 985 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260430 13:54:39.006670 984 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:39.007093 984 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:39.007234 984 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:39.017051 984 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:39.017266 984 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.105.1
I20260430 13:54:39.029636 984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.105.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.105.1
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:39.031317 984 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:39.033360 984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:39.043809 991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:39.044595 992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:39.051752 994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:39.055331 984 server_base.cc:1061] running on GCE node
I20260430 13:54:39.056212 984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:39.057457 984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:39.058761 984 hybrid_clock.cc:648] HybridClock initialized: now 1777557279058652 us; error 55 us; skew 500 ppm
I20260430 13:54:39.059149 984 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:39.062288 984 webserver.cc:492] Webserver started at http://127.0.105.1:45903/ using document root <none> and password file <none>
I20260430 13:54:39.063110 984 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:39.063221 984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:39.063575 984 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 13:54:39.066342 984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data/instance:
uuid: "72423ef5b9b54d4690d5050d879590d7"
format_stamp: "Formatted at 2026-04-30 13:54:39 on dist-test-slave-fmnx"
I20260430 13:54:39.067293 984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/wal/instance:
uuid: "72423ef5b9b54d4690d5050d879590d7"
format_stamp: "Formatted at 2026-04-30 13:54:39 on dist-test-slave-fmnx"
I20260430 13:54:39.073648 984 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260430 13:54:39.077472 1000 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:39.079941 984 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260430 13:54:39.080148 984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/wal
uuid: "72423ef5b9b54d4690d5050d879590d7"
format_stamp: "Formatted at 2026-04-30 13:54:39 on dist-test-slave-fmnx"
I20260430 13:54:39.080371 984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:39.102993 984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:39.103986 984 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:39.104257 984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:39.105360 984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 13:54:39.107658 984 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 13:54:39.107795 984 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:39.107935 984 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 13:54:39.108018 984 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:39.178457 984 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.1:44493
I20260430 13:54:39.178506 1112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.1:44493 every 8 connection(s)
I20260430 13:54:39.180156 984 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/data/info.pb
I20260430 13:54:39.182121 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 984
I20260430 13:54:39.182363 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-0/wal/instance
I20260430 13:54:39.193482 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.105.2:0
--local_ip_for_outbound_sockets=127.0.105.2
--webserver_interface=127.0.105.2
--webserver_port=0
--tserver_master_addrs=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260430 13:54:39.202538 1115 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:54:39.203130 1115 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:39.204478 1115 heartbeater.cc:507] Master 127.0.105.61:38801 requested a full tablet report, sending...
I20260430 13:54:39.208814 807 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:54:39.210075 1113 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:39.210213 1113 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:39.210794 1113 heartbeater.cc:507] Master 127.0.105.60:41159 requested a full tablet report, sending...
I20260430 13:54:39.212241 1114 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:39.212415 1114 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:39.212637 877 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:54:39.212945 1114 heartbeater.cc:507] Master 127.0.105.62:35887 requested a full tablet report, sending...
I20260430 13:54:39.214448 877 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.105.1:56603
I20260430 13:54:39.215008 737 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:54:39.236496 953 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:39.236629 953 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: loading cluster ID for follower catalog manager: success
I20260430 13:54:39.240267 953 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: acquiring CA information for follower catalog manager: success
I20260430 13:54:39.243250 953 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:54:39.339587 1124 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f962b03e872d4cb990156ed62ea6afd3)
I20260430 13:54:39.339895 1125 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f962b03e872d4cb990156ed62ea6afd3)
I20260430 13:54:39.339923 1124 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.340149 1125 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.341871 1125 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:39.341913 746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001" is_pre_election: true
I20260430 13:54:39.342144 746 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f615c6cf70da4e8f8c4dac485b923dee in term 1.
I20260430 13:54:39.342787 792 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee; no voters:
I20260430 13:54:39.341920 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee" is_pre_election: true
I20260430 13:54:39.343230 1125 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 13:54:39.343365 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 1.
I20260430 13:54:39.343700 1124 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:39.343730 886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:54:39.343986 887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:54:39.344013 724 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee; no voters:
I20260430 13:54:39.344457 1125 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Starting leader election (detected failure of leader f962b03e872d4cb990156ed62ea6afd3)
I20260430 13:54:39.344475 1124 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 13:54:39.344615 1124 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f962b03e872d4cb990156ed62ea6afd3)
I20260430 13:54:39.344636 1125 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 1 FOLLOWER]: Advancing to term 2
I20260430 13:54:39.344791 1124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 1 FOLLOWER]: Advancing to term 2
I20260430 13:54:39.348254 1125 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.348281 1124 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.349602 746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001"
I20260430 13:54:39.349689 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee"
I20260430 13:54:39.349879 746 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f615c6cf70da4e8f8c4dac485b923dee in current term 2: Already voted for candidate d392b5529f4245c397bd78166d0bc001 in this term.
I20260430 13:54:39.349949 815 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d392b5529f4245c397bd78166d0bc001 in current term 2: Already voted for candidate f615c6cf70da4e8f8c4dac485b923dee in this term.
I20260430 13:54:39.350975 1124 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 2 election: Requested vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:39.351346 887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:54:39.351796 886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:54:39.352036 1125 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 2 election: Requested vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:39.352046 722 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d392b5529f4245c397bd78166d0bc001; no voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3
I20260430 13:54:39.352555 1124 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 13:54:39.353089 815 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 2 FOLLOWER]: Rejecting Update request from peer f962b03e872d4cb990156ed62ea6afd3 for earlier term 1. Current term is 2. Ops: []
I20260430 13:54:39.353072 792 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee; no voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3
I20260430 13:54:39.353533 1125 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 13:54:39.353754 1126 consensus_queue.cc:1059] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 13:54:39.354378 1127 raft_consensus.cc:3055] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 13:54:39.354457 1127 raft_consensus.cc:740] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Running, Role: LEADER
I20260430 13:54:39.354789 1127 consensus_queue.cc:260] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.355261 1127 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 1 FOLLOWER]: Advancing to term 2
W20260430 13:54:39.577785 1123 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:39.578116 1123 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:39.578239 1123 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:39.587567 1123 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:39.587779 1123 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.105.2
I20260430 13:54:39.598800 1123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.105.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.105.2
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:39.600797 1123 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:39.602772 1123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:39.613083 1132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:39.613121 1133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:39.614315 1135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:39.615227 1123 server_base.cc:1061] running on GCE node
I20260430 13:54:39.616050 1123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:39.617103 1123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:39.618324 1123 hybrid_clock.cc:648] HybridClock initialized: now 1777557279618274 us; error 39 us; skew 500 ppm
I20260430 13:54:39.618649 1123 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:39.621065 1123 webserver.cc:492] Webserver started at http://127.0.105.2:44601/ using document root <none> and password file <none>
I20260430 13:54:39.621837 1123 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:39.621932 1123 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:39.622259 1123 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 13:54:39.624786 1123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data/instance:
uuid: "fde79c28b7c94c3f9466dcc8516fbad5"
format_stamp: "Formatted at 2026-04-30 13:54:39 on dist-test-slave-fmnx"
I20260430 13:54:39.625572 1123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/wal/instance:
uuid: "fde79c28b7c94c3f9466dcc8516fbad5"
format_stamp: "Formatted at 2026-04-30 13:54:39 on dist-test-slave-fmnx"
I20260430 13:54:39.631343 1123 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260430 13:54:39.635219 1141 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:39.636968 1123 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 13:54:39.637151 1123 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/wal
uuid: "fde79c28b7c94c3f9466dcc8516fbad5"
format_stamp: "Formatted at 2026-04-30 13:54:39 on dist-test-slave-fmnx"
I20260430 13:54:39.637359 1123 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:39.673280 1123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:39.674209 1123 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:39.674496 1123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:39.675534 1123 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 13:54:39.677343 1123 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 13:54:39.677451 1123 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:39.677564 1123 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 13:54:39.677636 1123 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:39.715991 1123 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.2:35227
I20260430 13:54:39.716226 1253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.2:35227 every 8 connection(s)
I20260430 13:54:39.717247 1113 heartbeater.cc:499] Master 127.0.105.60:41159 was elected leader, sending a full tablet report...
I20260430 13:54:39.717996 1123 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/data/info.pb
I20260430 13:54:39.718454 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 1123
I20260430 13:54:39.718639 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-1/wal/instance
I20260430 13:54:39.745911 1255 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:39.746294 1255 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:39.747387 1255 heartbeater.cc:507] Master 127.0.105.62:35887 requested a full tablet report, sending...
I20260430 13:54:39.748682 1254 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:39.748806 1256 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:54:39.748862 1254 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:39.748956 1256 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:39.749168 737 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:39.749481 1254 heartbeater.cc:507] Master 127.0.105.60:41159 requested a full tablet report, sending...
I20260430 13:54:39.749506 1256 heartbeater.cc:507] Master 127.0.105.61:38801 requested a full tablet report, sending...
I20260430 13:54:39.750458 807 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:39.750476 877 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:39.892395 1264 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:39.892573 1264 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.893239 1264 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:39.893859 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee" is_pre_election: true
I20260430 13:54:39.893839 886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:54:39.894076 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 2.
I20260430 13:54:39.894137 886 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 2.
I20260430 13:54:39.894508 724 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee; no voters:
I20260430 13:54:39.894788 1264 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 13:54:39.894891 1264 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 13:54:39.895460 1264 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 2 FOLLOWER]: Advancing to term 3
I20260430 13:54:39.897464 1264 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.898098 1264 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 3 election: Requested vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:39.898633 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee"
I20260430 13:54:39.898845 886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:54:39.898978 815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 2 FOLLOWER]: Advancing to term 3
I20260430 13:54:39.899046 886 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 2 FOLLOWER]: Advancing to term 3
I20260430 13:54:39.901124 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 3.
I20260430 13:54:39.901247 886 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 3.
I20260430 13:54:39.901664 724 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee; no voters:
I20260430 13:54:39.902024 1264 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 13:54:39.902402 1264 raft_consensus.cc:697] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 3 LEADER]: Becoming Leader. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Running, Role: LEADER
I20260430 13:54:39.902859 1264 consensus_queue.cc:237] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:39.904695 1265 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d392b5529f4245c397bd78166d0bc001. Latest consensus state: current_term: 3 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:39.905072 1265 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:39.905723 1267 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:54:39.907661 1267 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:54:39.908484 1267 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:39.908557 1267 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:54:39.910444 1267 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:54:39.911329 1267 catalog_manager.cc:6055] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Loaded TSK: 0
I20260430 13:54:39.912295 1267 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:54:39.913059 1267 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:54:39.913820 1267 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:54:39.932682 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.105.3:0
--local_ip_for_outbound_sockets=127.0.105.3
--webserver_interface=127.0.105.3
--webserver_port=0
--tserver_master_addrs=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260430 13:54:40.117578 934 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:54:40.253124 737 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.105.2:50773
W20260430 13:54:40.296257 1268 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:40.296568 1268 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:40.296697 1268 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:40.306352 1268 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:40.306591 1268 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.105.3
I20260430 13:54:40.317464 1268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.105.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.105.3
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:40.319074 1268 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:40.320912 1268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:40.330978 1274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:40.331003 1273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:40.331784 1276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:40.332458 1268 server_base.cc:1061] running on GCE node
I20260430 13:54:40.333076 1268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:40.334079 1268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:40.335289 1268 hybrid_clock.cc:648] HybridClock initialized: now 1777557280335234 us; error 43 us; skew 500 ppm
I20260430 13:54:40.335584 1268 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:40.337752 1268 webserver.cc:492] Webserver started at http://127.0.105.3:41707/ using document root <none> and password file <none>
I20260430 13:54:40.338595 1268 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:40.338707 1268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:40.339046 1268 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 13:54:40.342136 1268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data/instance:
uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f"
format_stamp: "Formatted at 2026-04-30 13:54:40 on dist-test-slave-fmnx"
I20260430 13:54:40.343101 1268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/wal/instance:
uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f"
format_stamp: "Formatted at 2026-04-30 13:54:40 on dist-test-slave-fmnx"
I20260430 13:54:40.348996 1268 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260430 13:54:40.352929 1282 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:40.354436 1268 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 13:54:40.354607 1268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/wal
uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f"
format_stamp: "Formatted at 2026-04-30 13:54:40 on dist-test-slave-fmnx"
I20260430 13:54:40.354885 1268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:40.368906 1268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:40.369800 1268 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:40.370062 1268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:40.371493 1268 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 13:54:40.373227 1268 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 13:54:40.373313 1268 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:40.373428 1268 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 13:54:40.373503 1268 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:40.410048 1268 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.3:46429
I20260430 13:54:40.410096 1394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.3:46429 every 8 connection(s)
I20260430 13:54:40.411460 1268 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/data/info.pb
I20260430 13:54:40.421470 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 1268
I20260430 13:54:40.421677 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/ts-2/wal/instance
I20260430 13:54:40.435771 1395 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:40.436102 1397 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:54:40.436175 1395 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:40.436331 1397 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:40.437362 1395 heartbeater.cc:507] Master 127.0.105.60:41159 requested a full tablet report, sending...
I20260430 13:54:40.437388 1397 heartbeater.cc:507] Master 127.0.105.61:38801 requested a full tablet report, sending...
I20260430 13:54:40.438129 1396 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:40.438269 1396 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:40.438684 1396 heartbeater.cc:507] Master 127.0.105.62:35887 requested a full tablet report, sending...
I20260430 13:54:40.439044 807 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:40.439550 737 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:40.439729 877 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:40.440506 737 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.105.3:33741
I20260430 13:54:40.449115 420 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 13:54:40.466560 1412 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:40.466837 1412 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:40.467623 1412 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
I20260430 13:54:40.468281 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee" is_pre_election: true
I20260430 13:54:40.468510 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f962b03e872d4cb990156ed62ea6afd3 in term 3.
I20260430 13:54:40.468600 746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001" is_pre_election: true
I20260430 13:54:40.469030 864 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters:
I20260430 13:54:40.469377 1412 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 13:54:40.469499 1412 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 13:54:40.469581 1412 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 3 FOLLOWER]: Advancing to term 4
I20260430 13:54:40.471486 1412 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:40.472424 746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001"
I20260430 13:54:40.472479 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee"
I20260430 13:54:40.472653 815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 3 FOLLOWER]: Advancing to term 4
I20260430 13:54:40.474689 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f962b03e872d4cb990156ed62ea6afd3 in term 4.
I20260430 13:54:40.475016 1412 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 4 election: Requested vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
I20260430 13:54:40.475245 864 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001
I20260430 13:54:40.475651 1412 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 13:54:40.476109 1412 raft_consensus.cc:697] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 LEADER]: Becoming Leader. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Running, Role: LEADER
I20260430 13:54:40.476465 1412 consensus_queue.cc:237] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:40.477947 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f962b03e872d4cb990156ed62ea6afd3. Latest consensus state: current_term: 4 leader_uuid: "f962b03e872d4cb990156ed62ea6afd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:40.478224 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:40.482424 737 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_9240d1b8bb1f4c99884d30bb311335d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.483852 736 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_646011822dc5429cb2f256da808338bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.487358 1418 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:54:40.490399 1418 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:54:40.491812 1418 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:40.491951 1418 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
W20260430 13:54:40.492164 737 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9240d1b8bb1f4c99884d30bb311335d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 13:54:40.493805 1418 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:54:40.494859 1418 catalog_manager.cc:6055] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Loaded TSK: 0
I20260430 13:54:40.494980 735 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_38caf5dea0244ad6ba8033a82343784b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.495937 1418 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:54:40.496073 734 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_fe99e22f9c214f8e9082c50a02935393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.497687 1418 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:54:40.498703 1418 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:54:40.501775 886 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 LEADER]: Rejecting Update request from peer d392b5529f4245c397bd78166d0bc001 for earlier term 3. Current term is 4. Ops: []
I20260430 13:54:40.502014 815 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 4 FOLLOWER]: Rejecting Update request from peer d392b5529f4245c397bd78166d0bc001 for earlier term 3. Current term is 4. Ops: []
I20260430 13:54:40.502861 1423 consensus_queue.cc:1059] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 13:54:40.504297 733 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_535de9acd8c747cba6ae09ea257f82c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.504951 1425 consensus_queue.cc:1059] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 13:54:40.505250 1425 consensus_queue.cc:1243] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Peer f962b03e872d4cb990156ed62ea6afd3 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260430 13:54:40.507720 1423 raft_consensus.cc:3055] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 13:54:40.507901 1423 raft_consensus.cc:740] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Running, Role: LEADER
I20260430 13:54:40.508309 1423 consensus_queue.cc:260] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:40.508936 1423 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 3 FOLLOWER]: Advancing to term 4
W20260430 13:54:40.517442 1406 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.62:35887)
I20260430 13:54:40.525033 876 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35578:
name: "default.Table_5e7ce2408b714cc9b44c00d9d03919d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.529369 877 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35578:
name: "default.Table_cc4a2daeb0de4cc182167aa213911136"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.538048 874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35578:
name: "default.Table_b90a5e6d6e8b44748820ab72580fa9e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.538293 875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35578:
name: "default.Table_1d395d40fbbc4690af86be9b01485cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:40.538071 873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35578:
name: "default.Table_c83031f7e5634413a1d01b8bce95a9d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 13:54:40.551254 876 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5e7ce2408b714cc9b44c00d9d03919d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 13:54:40.643365 420 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu pid 843
I20260430 13:54:40.645012 907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260430 13:54:40.721172 1114 heartbeater.cc:499] Master 127.0.105.62:35887 was elected leader, sending a full tablet report...
I20260430 13:54:40.755765 1255 heartbeater.cc:499] Master 127.0.105.62:35887 was elected leader, sending a full tablet report...
I20260430 13:54:40.829985 420 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu with pid 843
W20260430 13:54:40.868273 1410 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 13:54:40.862499 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.60:41159 (local address 127.0.0.1:35590)
0430 13:54:40.863262 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0430 13:54:40.863283 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0430 13:54:40.863456 (+ 173us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 13:54:40.866623 (+ 3167us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":488}
I20260430 13:54:40.942896 1396 heartbeater.cc:499] Master 127.0.105.62:35887 was elected leader, sending a full tablet report...
W20260430 13:54:40.964982 1395 heartbeater.cc:646] Failed to heartbeat to 127.0.105.60:41159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:41.098276 737 catalog_manager.cc:2174] created HMS catalog entry for table [id=f3ef536ae2384c829da92f20ac2d7c9c]
W20260430 13:54:41.100303 737 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: replica d392b5529f4245c397bd78166d0bc001 is not leader of this config: current role FOLLOWER
W20260430 13:54:41.101790 736 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_646011822dc5429cb2f256da808338bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260430 13:54:41.101795 735 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_38caf5dea0244ad6ba8033a82343784b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260430 13:54:41.103015 734 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fe99e22f9c214f8e9082c50a02935393 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260430 13:54:41.104108 733 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_535de9acd8c747cba6ae09ea257f82c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 13:54:41.106828 1455 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:41.107071 1455 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:41.107887 1455 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:41.108721 746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001" is_pre_election: true
I20260430 13:54:41.109023 746 raft_consensus.cc:2410] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f615c6cf70da4e8f8c4dac485b923dee for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 13:54:41.110706 1454 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:41.110948 1454 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:41.111755 1454 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:41.112273 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee" is_pre_election: true
I20260430 13:54:41.112486 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 4.
I20260430 13:54:41.113214 724 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee; no voters:
I20260430 13:54:41.124452 1454 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 13:54:41.124624 1454 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 13:54:41.124715 1454 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 4 FOLLOWER]: Advancing to term 5
W20260430 13:54:41.126883 792 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159): Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:41.127166 792 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee; no voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3
I20260430 13:54:41.127537 1455 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 13:54:41.129945 1454 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:41.133291 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee"
I20260430 13:54:41.133515 815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 4 FOLLOWER]: Advancing to term 5
I20260430 13:54:41.136468 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 5.
I20260430 13:54:41.137045 724 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee; no voters:
I20260430 13:54:41.168442 1459 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 13:54:41.172966 1454 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 5 election: Requested vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:41.173825 1459 raft_consensus.cc:697] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 LEADER]: Becoming Leader. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Running, Role: LEADER
I20260430 13:54:41.174196 1459 consensus_queue.cc:237] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
W20260430 13:54:41.190768 722 leader_election.cc:336] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159): Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
W20260430 13:54:41.192229 722 leader_election.cc:336] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159): Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:41.193845 1454 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d392b5529f4245c397bd78166d0bc001. Latest consensus state: current_term: 5 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:41.194070 1454 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:41.330902 1113 heartbeater.cc:499] Master 127.0.105.60:41159 was elected leader, sending a full tablet report...
I20260430 13:54:41.331333 1254 heartbeater.cc:499] Master 127.0.105.60:41159 was elected leader, sending a full tablet report...
W20260430 13:54:41.366917 1254 heartbeater.cc:646] Failed to heartbeat to 127.0.105.60:41159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
W20260430 13:54:41.374941 1113 heartbeater.cc:646] Failed to heartbeat to 127.0.105.60:41159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111)
I20260430 13:54:41.569058 815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 5 FOLLOWER]: Refusing update from remote peer d392b5529f4245c397bd78166d0bc001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260430 13:54:41.569859 1454 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 13:54:41.589294 1455 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d392b5529f4245c397bd78166d0bc001. Latest consensus state: current_term: 5 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:41.589561 1455 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
W20260430 13:54:41.602051 722 consensus_peers.cc:597] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 -> Peer f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159): Couldn't send request to peer f962b03e872d4cb990156ed62ea6afd3. Status: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260430 13:54:41.603926 1460 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:41.604206 1460 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:41.612988 1455 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:41.613229 1455 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:41.614018 1470 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:41.614356 1470 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:41.634114 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--webserver_interface=127.0.105.60
--webserver_port=43565
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260430 13:54:41.789191 737 catalog_manager.cc:2184] deleted orphaned HMS catalog entry for table [id=f3ef536ae2384c829da92f20ac2d7c9c]
I20260430 13:54:41.789849 737 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:60940 (request call id 1) took 1308 ms. Trace:
I20260430 13:54:41.789961 737 rpcz_store.cc:276] 0430 13:54:40.481697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 13:54:40.481816 (+ 119us) service_pool.cc:225] Handling call
0430 13:54:40.492426 (+ 10610us) catalog_manager.cc:2101] Acquired catalog manager lock
0430 13:54:40.493631 (+ 1205us) catalog_manager.cc:2147] Created new table and tablet info
0430 13:54:41.098244 (+604613us) catalog_manager.cc:2173] Created new table in HMS catalog
0430 13:54:41.100045 (+ 1801us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:41.100050 (+ 5us) write_op.cc:454] Released schema lock
0430 13:54:41.100432 (+ 382us) catalog_manager.cc:2181] Rolling back HMS table creation
0430 13:54:41.789785 (+689353us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 13:54:41.099827 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica d392b5529f4245c397bd78166d0bc001 is not leader of this config: current role FOLLOWER)
0430 13:54:41.099842 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:41.099851 (+ 9us) write_op.cc:454] Released schema lock
0430 13:54:41.099856 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":5478,"Hive Metastore.run_cpu_time_us":1036,"Hive Metastore.run_wall_time_us":1287115,"child_traces":[["op",{}]]}
W20260430 13:54:41.791635 1406 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.60:41159)
I20260430 13:54:41.840117 735 catalog_manager.cc:2174] created HMS catalog entry for table [id=1e3228304b9349cf86cc688efee611f0]
I20260430 13:54:41.848567 735 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:60940 (request call id 3) took 1354 ms. Trace:
I20260430 13:54:41.848726 735 rpcz_store.cc:276] 0430 13:54:40.494620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 13:54:40.494687 (+ 67us) service_pool.cc:225] Handling call
0430 13:54:41.102041 (+607354us) catalog_manager.cc:2101] Acquired catalog manager lock
0430 13:54:41.102544 (+ 503us) catalog_manager.cc:2147] Created new table and tablet info
0430 13:54:41.840041 (+737497us) catalog_manager.cc:2173] Created new table in HMS catalog
0430 13:54:41.847854 (+ 7813us) catalog_manager.cc:2207] Wrote table and tablets to system table
0430 13:54:41.848315 (+ 461us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0430 13:54:41.848509 (+ 194us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 13:54:41.841127 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 13:54:41.841342 (+ 215us) write_op.cc:432] Acquiring schema lock in shared mode
0430 13:54:41.841351 (+ 9us) write_op.cc:435] Acquired schema lock
0430 13:54:41.841354 (+ 3us) tablet.cc:662] Decoding operations
0430 13:54:41.841564 (+ 210us) write_op.cc:620] Acquiring the partition lock for write op
0430 13:54:41.841582 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0430 13:54:41.841600 (+ 18us) tablet.cc:685] Acquiring locks for 2 operations
0430 13:54:41.841723 (+ 123us) tablet.cc:701] Row locks acquired
0430 13:54:41.841725 (+ 2us) write_op.cc:260] PREPARE: finished
0430 13:54:41.841788 (+ 63us) write_op.cc:270] Start()
0430 13:54:41.841834 (+ 46us) write_op.cc:276] Timestamp: P: 1777557281841758 usec, L: 0
0430 13:54:41.841844 (+ 10us) op_driver.cc:348] REPLICATION: starting
0430 13:54:41.842144 (+ 300us) log.cc:844] Serialized 432 byte log entry
0430 13:54:41.846070 (+ 3926us) op_driver.cc:464] REPLICATION: finished
0430 13:54:41.846687 (+ 617us) write_op.cc:301] APPLY: starting
0430 13:54:41.846707 (+ 20us) tablet.cc:1367] starting BulkCheckPresence
0430 13:54:41.846776 (+ 69us) tablet.cc:1370] finished BulkCheckPresence
0430 13:54:41.846807 (+ 31us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 13:54:41.846970 (+ 163us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 13:54:41.847015 (+ 45us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 13:54:41.847022 (+ 7us) write_op.cc:312] APPLY: finished
0430 13:54:41.847123 (+ 101us) log.cc:844] Serialized 28 byte log entry
0430 13:54:41.847467 (+ 344us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:41.847675 (+ 208us) write_op.cc:454] Released schema lock
0430 13:54:41.847685 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":686483,"Hive Metastore.run_cpu_time_us":332,"Hive Metastore.run_wall_time_us":50795,"spinlock_wait_cycles":7680,"child_traces":[["op",{"apply.queue_time_us":537,"num_ops":2,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":1195,"prepare.run_wall_time_us":1195,"raft.queue_time_us":1304,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":321,"replication_time_us":4188,"thread_start_us":1166,"threads_started":1}]]}
W20260430 13:54:41.852759 1406 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.60:41159)
I20260430 13:54:41.901227 736 catalog_manager.cc:2174] created HMS catalog entry for table [id=635b8a8981244f7d84fc2e58a88b9a02]
I20260430 13:54:41.911160 736 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:60940 (request call id 2) took 1427 ms. Trace:
I20260430 13:54:41.911336 736 rpcz_store.cc:276] 0430 13:54:40.483219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 13:54:40.483491 (+ 272us) service_pool.cc:225] Handling call
0430 13:54:41.101952 (+618461us) catalog_manager.cc:2101] Acquired catalog manager lock
0430 13:54:41.102535 (+ 583us) catalog_manager.cc:2147] Created new table and tablet info
0430 13:54:41.901155 (+798620us) catalog_manager.cc:2173] Created new table in HMS catalog
0430 13:54:41.910829 (+ 9674us) catalog_manager.cc:2207] Wrote table and tablets to system table
0430 13:54:41.910976 (+ 147us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0430 13:54:41.911099 (+ 123us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 13:54:41.902245 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 13:54:41.902402 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0430 13:54:41.902410 (+ 8us) write_op.cc:435] Acquired schema lock
0430 13:54:41.902413 (+ 3us) tablet.cc:662] Decoding operations
0430 13:54:41.902518 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0430 13:54:41.902534 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0430 13:54:41.902540 (+ 6us) tablet.cc:685] Acquiring locks for 2 operations
0430 13:54:41.902637 (+ 97us) tablet.cc:701] Row locks acquired
0430 13:54:41.902639 (+ 2us) write_op.cc:260] PREPARE: finished
0430 13:54:41.902690 (+ 51us) write_op.cc:270] Start()
0430 13:54:41.902732 (+ 42us) write_op.cc:276] Timestamp: P: 1777557281902660 usec, L: 0
0430 13:54:41.902735 (+ 3us) op_driver.cc:348] REPLICATION: starting
0430 13:54:41.903019 (+ 284us) log.cc:844] Serialized 432 byte log entry
0430 13:54:41.907292 (+ 4273us) op_driver.cc:464] REPLICATION: finished
0430 13:54:41.907493 (+ 201us) write_op.cc:301] APPLY: starting
0430 13:54:41.907515 (+ 22us) tablet.cc:1367] starting BulkCheckPresence
0430 13:54:41.907595 (+ 80us) tablet.cc:1370] finished BulkCheckPresence
0430 13:54:41.907646 (+ 51us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 13:54:41.907770 (+ 124us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 13:54:41.907827 (+ 57us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 13:54:41.907835 (+ 8us) write_op.cc:312] APPLY: finished
0430 13:54:41.908042 (+ 207us) log.cc:844] Serialized 28 byte log entry
0430 13:54:41.908122 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:41.908150 (+ 28us) write_op.cc:454] Released schema lock
0430 13:54:41.908159 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 13:54:41.908245 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:41.908249 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":737327,"Hive Metastore.run_cpu_time_us":311,"Hive Metastore.run_wall_time_us":60827,"mutex_wait_us":30,"spinlock_wait_cycles":4736,"child_traces":[["op",{"apply.queue_time_us":139,"apply.run_cpu_time_us":679,"apply.run_wall_time_us":744,"num_ops":2,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":957,"prepare.run_wall_time_us":957,"raft.queue_time_us":177,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":421,"replication_time_us":4516}]]}
I20260430 13:54:41.963100 734 catalog_manager.cc:2174] created HMS catalog entry for table [id=6c15f9d20dca40b4a95132cc17279a56]
I20260430 13:54:41.970301 734 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:60940 (request call id 4) took 1474 ms. Trace:
I20260430 13:54:41.970458 734 rpcz_store.cc:276] 0430 13:54:40.495740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 13:54:40.495799 (+ 59us) service_pool.cc:225] Handling call
0430 13:54:41.103155 (+607356us) catalog_manager.cc:2101] Acquired catalog manager lock
0430 13:54:41.103635 (+ 480us) catalog_manager.cc:2147] Created new table and tablet info
0430 13:54:41.963028 (+859393us) catalog_manager.cc:2173] Created new table in HMS catalog
0430 13:54:41.969953 (+ 6925us) catalog_manager.cc:2207] Wrote table and tablets to system table
0430 13:54:41.970138 (+ 185us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0430 13:54:41.970257 (+ 119us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 13:54:41.964221 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 13:54:41.964371 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0430 13:54:41.964379 (+ 8us) write_op.cc:435] Acquired schema lock
0430 13:54:41.964381 (+ 2us) tablet.cc:662] Decoding operations
0430 13:54:41.964493 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0430 13:54:41.964509 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0430 13:54:41.964514 (+ 5us) tablet.cc:685] Acquiring locks for 2 operations
0430 13:54:41.964600 (+ 86us) tablet.cc:701] Row locks acquired
0430 13:54:41.964602 (+ 2us) write_op.cc:260] PREPARE: finished
0430 13:54:41.964651 (+ 49us) write_op.cc:270] Start()
0430 13:54:41.964692 (+ 41us) write_op.cc:276] Timestamp: P: 1777557281964622 usec, L: 0
0430 13:54:41.964696 (+ 4us) op_driver.cc:348] REPLICATION: starting
0430 13:54:41.964921 (+ 225us) log.cc:844] Serialized 432 byte log entry
0430 13:54:41.969160 (+ 4239us) op_driver.cc:464] REPLICATION: finished
0430 13:54:41.969300 (+ 140us) write_op.cc:301] APPLY: starting
0430 13:54:41.969323 (+ 23us) tablet.cc:1367] starting BulkCheckPresence
0430 13:54:41.969376 (+ 53us) tablet.cc:1370] finished BulkCheckPresence
0430 13:54:41.969420 (+ 44us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 13:54:41.969614 (+ 194us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 13:54:41.969657 (+ 43us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 13:54:41.969664 (+ 7us) write_op.cc:312] APPLY: finished
0430 13:54:41.969753 (+ 89us) log.cc:844] Serialized 28 byte log entry
0430 13:54:41.969831 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:41.969855 (+ 24us) write_op.cc:454] Released schema lock
0430 13:54:41.969863 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 13:54:41.969950 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:41.969954 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":797154,"Hive Metastore.run_cpu_time_us":295,"Hive Metastore.run_wall_time_us":61765,"child_traces":[["op",{"apply.queue_time_us":48,"apply.run_cpu_time_us":654,"apply.run_wall_time_us":654,"num_ops":2,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":834,"prepare.run_wall_time_us":836,"raft.queue_time_us":951,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":280,"replication_time_us":4429}]]}
I20260430 13:54:42.060395 733 catalog_manager.cc:2174] created HMS catalog entry for table [id=bae9523e823c4703ac8463cb06496d1a]
I20260430 13:54:42.070058 733 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:60940 (request call id 5) took 1566 ms. Trace:
I20260430 13:54:42.070235 733 rpcz_store.cc:276] 0430 13:54:40.503950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 13:54:40.504005 (+ 55us) service_pool.cc:225] Handling call
0430 13:54:41.104225 (+600220us) catalog_manager.cc:2101] Acquired catalog manager lock
0430 13:54:41.104717 (+ 492us) catalog_manager.cc:2147] Created new table and tablet info
0430 13:54:42.060320 (+955603us) catalog_manager.cc:2173] Created new table in HMS catalog
0430 13:54:42.069728 (+ 9408us) catalog_manager.cc:2207] Wrote table and tablets to system table
0430 13:54:42.069876 (+ 148us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0430 13:54:42.070002 (+ 126us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 13:54:42.061324 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 13:54:42.061462 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0430 13:54:42.061469 (+ 7us) write_op.cc:435] Acquired schema lock
0430 13:54:42.061471 (+ 2us) tablet.cc:662] Decoding operations
0430 13:54:42.061579 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0430 13:54:42.061596 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0430 13:54:42.061601 (+ 5us) tablet.cc:685] Acquiring locks for 2 operations
0430 13:54:42.061683 (+ 82us) tablet.cc:701] Row locks acquired
0430 13:54:42.061686 (+ 3us) write_op.cc:260] PREPARE: finished
0430 13:54:42.061724 (+ 38us) write_op.cc:270] Start()
0430 13:54:42.061771 (+ 47us) write_op.cc:276] Timestamp: P: 1777557282061700 usec, L: 0
0430 13:54:42.061775 (+ 4us) op_driver.cc:348] REPLICATION: starting
0430 13:54:42.061985 (+ 210us) log.cc:844] Serialized 432 byte log entry
0430 13:54:42.065917 (+ 3932us) op_driver.cc:464] REPLICATION: finished
0430 13:54:42.066025 (+ 108us) write_op.cc:301] APPLY: starting
0430 13:54:42.066065 (+ 40us) tablet.cc:1367] starting BulkCheckPresence
0430 13:54:42.066132 (+ 67us) tablet.cc:1370] finished BulkCheckPresence
0430 13:54:42.066162 (+ 30us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 13:54:42.066368 (+ 206us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 13:54:42.066407 (+ 39us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 13:54:42.066414 (+ 7us) write_op.cc:312] APPLY: finished
0430 13:54:42.066514 (+ 100us) log.cc:844] Serialized 28 byte log entry
0430 13:54:42.066591 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:42.066619 (+ 28us) write_op.cc:454] Released schema lock
0430 13:54:42.066626 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 13:54:42.066709 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0430 13:54:42.066712 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":857904,"Hive Metastore.run_cpu_time_us":310,"Hive Metastore.run_wall_time_us":97451,"child_traces":[["op",{"apply.queue_time_us":38,"apply.run_cpu_time_us":678,"apply.run_wall_time_us":678,"num_ops":2,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":797,"prepare.run_wall_time_us":796,"raft.queue_time_us":192,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":298,"replication_time_us":4100}]]}
I20260430 13:54:42.071569 1463 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:54:42.074173 1463 catalog_manager.cc:679] Loaded metadata for table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0]
I20260430 13:54:42.074743 1463 catalog_manager.cc:679] Loaded metadata for table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02]
I20260430 13:54:42.075173 1463 catalog_manager.cc:679] Loaded metadata for table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56]
I20260430 13:54:42.075577 1463 catalog_manager.cc:679] Loaded metadata for table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a]
I20260430 13:54:42.077433 1463 tablet_loader.cc:96] loaded metadata for tablet 01949d383b2c44d583e43997ba847d49 (table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a])
I20260430 13:54:42.077836 1463 tablet_loader.cc:96] loaded metadata for tablet 349ee22fa537461ba9f48a3121f2c39c (table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0])
I20260430 13:54:42.078131 1463 tablet_loader.cc:96] loaded metadata for tablet 85be806c4e1b404d9d3e30cde7df0286 (table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56])
I20260430 13:54:42.078430 1463 tablet_loader.cc:96] loaded metadata for tablet a5821dad44c048548e5764a0dc87f967 (table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02])
I20260430 13:54:42.078851 1463 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:54:42.081882 1463 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:42.082036 1463 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:54:42.083916 1463 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:54:42.085300 1463 catalog_manager.cc:6055] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Loaded TSK: 0
I20260430 13:54:42.086041 1463 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:54:42.087126 1463 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:54:42.088169 1463 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
W20260430 13:54:42.101284 722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.60:41159: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20260430 13:54:42.109030 735 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_cc4a2daeb0de4cc182167aa213911136"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.109359 734 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_b90a5e6d6e8b44748820ab72580fa9e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.109696 737 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_5e7ce2408b714cc9b44c00d9d03919d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.109862 736 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_1d395d40fbbc4690af86be9b01485cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.110265 733 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_c83031f7e5634413a1d01b8bce95a9d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.111145 731 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_9240d1b8bb1f4c99884d30bb311335d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.111862 730 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60940:
table { table_name: "default.Table_38caf5dea0244ad6ba8033a82343784b" } new_table_name: "default.Table_4839c82734a34812b58582d1a7da4255" modify_external_catalogs: true
I20260430 13:54:42.111721 728 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "default.Table_106dd7fc76304c5eadea2552b3735996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.111939 729 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60940:
table { table_name: "default.Table_fe99e22f9c214f8e9082c50a02935393" } new_table_name: "default.Table_da590a7148884972933145e283c0b67f" modify_external_catalogs: true
I20260430 13:54:42.114086 732 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60940:
table { table_name: "default.Table_646011822dc5429cb2f256da808338bc" } new_table_name: "default.Table_5928dba2f92a45df98b70003873a6ebe" modify_external_catalogs: true
F20260430 13:54:42.123394 952 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 7 received events: NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_9240d1b8bb1f4c99884d30bb311335d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_9240d1b8bb1f4c99884d30bb311335d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9240d1b8bb1f4c99884d30bb311335d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_9240d1b8bb1f4c99884d30bb311335d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9240d1b8bb1f4c99884d30bb311335d1\",\"kudu.table_id\":\"f3ef536ae2384c829da92f20ac2d7c9c\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_5e7ce2408b714cc9b44c00d9d03919d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_5e7ce2408b714cc9b44c00d9d03919d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5e7ce2408b714cc9b44c00d9d03919d8\",\"kudu.table_id\":\"5aa844dc225b4ce09bb01e23e5321864\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fdaf594adcd google::LogMessage::Fail() at ??:0
@ 0x7fdaf594eb93 google::LogMessage::SendToLog() at ??:0
W20260430 13:54:42.128038 1471 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:42.128487 1471 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:42.128599 1471 flags.cc:432] Enabled unsafe flag: --never_fsync=true
@ 0x7fdaf594a7cc google::LogMessage::Flush() at ??:0
@ 0x7fdaf594aa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fdb0c3ff574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fdb0c3fde21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
W20260430 13:54:42.143942 1471 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:42.144054 1471 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:42.144137 1471 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:42.144194 1471 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
@ 0x7fdb0c405a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fdb0c405919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fdaf86e3529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdafbcb16db start_thread at ??:0
@ 0x7fdaf47fb61f clone at ??:0
I20260430 13:54:42.161728 1471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:42.164165 1471 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:42.166826 1471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:42.188416 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:42.188515 1489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:42.202698 1493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:42.202952 1471 server_base.cc:1061] running on GCE node
I20260430 13:54:42.204337 1471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:42.206596 1471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:42.207850 1471 hybrid_clock.cc:648] HybridClock initialized: now 1777557282207788 us; error 56 us; skew 500 ppm
I20260430 13:54:42.208364 1471 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:42.211421 1471 webserver.cc:492] Webserver started at http://127.0.105.60:43565/ using document root <none> and password file <none>
I20260430 13:54:42.212553 1471 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:42.212702 1471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:42.220810 1471 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.004s
I20260430 13:54:42.224370 1502 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:42.226117 1471 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260430 13:54:42.226320 1471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:42.227222 1471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:42.246359 1471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:42.247329 1471 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:42.247628 1471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:42.275835 1471 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:41159
I20260430 13:54:42.275863 1553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:41159 every 8 connection(s)
I20260430 13:54:42.277493 1471 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
I20260430 13:54:42.279757 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 1471
I20260430 13:54:42.288282 1554 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:54:42.293357 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap starting.
I20260430 13:54:42.305385 1554 log.cc:826] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:42.318177 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260430 13:54:42.318920 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap complete.
I20260430 13:54:42.326416 1554 raft_consensus.cc:359] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:42.327737 1554 raft_consensus.cc:740] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Initialized, Role: FOLLOWER
I20260430 13:54:42.328579 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:42.329449 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:42.329772 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:42.330967 1554 sys_catalog.cc:565] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:54:42.359769 1572 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:42.360002 1572 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: loading cluster ID for follower catalog manager: success
I20260430 13:54:42.364254 1572 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: acquiring CA information for follower catalog manager: success
I20260430 13:54:42.367233 1572 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:54:42.379196 1254 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:42.379441 1254 heartbeater.cc:499] Master 127.0.105.60:41159 was elected leader, sending a full tablet report...
I20260430 13:54:42.381799 1519 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:58053; Asking this server to re-register.
I20260430 13:54:42.382880 1254 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:42.384879 1519 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:42.405822 1113 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:42.406088 1113 heartbeater.cc:499] Master 127.0.105.60:41159 was elected leader, sending a full tablet report...
I20260430 13:54:42.407200 1519 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:59461; Asking this server to re-register.
I20260430 13:54:42.407851 1113 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:42.409286 1519 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
W20260430 13:54:42.430612 1406 connection.cc:570] client connection to 127.0.105.62:35887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260430 13:54:42.430635 1143 connection.cc:570] client connection to 127.0.105.62:35887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260430 13:54:42.432044 1002 connection.cc:570] client connection to 127.0.105.62:35887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260430 13:54:42.434798 1575 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 13:54:42.340435 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.62:35887 (local address 127.0.0.1:60944)
0430 13:54:42.355342 (+ 14907us) negotiation.cc:107] Waiting for socket to connect
0430 13:54:42.355412 (+ 70us) client_negotiation.cc:175] Beginning negotiation
0430 13:54:42.355938 (+ 526us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 13:54:42.433048 (+ 77110us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":14598,"thread_start_us":12448,"threads_started":1}
I20260430 13:54:42.438544 1471 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20260430 13:54:42.444576 1471 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:42.502475 1395 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:42.503614 1519 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:54911; Asking this server to re-register.
I20260430 13:54:42.504248 1395 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:42.506269 1519 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:42.567929 1581 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d392b5529f4245c397bd78166d0bc001)
I20260430 13:54:42.568132 1581 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:42.568926 1581 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
W20260430 13:54:42.571727 792 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:54:42.577381 1529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:54:42.578063 1529 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f615c6cf70da4e8f8c4dac485b923dee in term 4.
I20260430 13:54:42.578696 792 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001
I20260430 13:54:42.579062 1581 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 13:54:42.579187 1581 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 5 FOLLOWER]: Starting leader election (detected failure of leader d392b5529f4245c397bd78166d0bc001)
I20260430 13:54:42.579285 1581 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 5 FOLLOWER]: Advancing to term 6
I20260430 13:54:42.581954 1581 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:42.582520 1581 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 6 election: Requested vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:42.583431 1529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:54:42.583654 1529 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 4 FOLLOWER]: Advancing to term 6
W20260430 13:54:42.583976 792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260430 13:54:42.585181 792 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:54:42.589673 1529 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f615c6cf70da4e8f8c4dac485b923dee in term 6.
I20260430 13:54:42.590262 792 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001
I20260430 13:54:42.590629 1581 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Leader election won for term 6
I20260430 13:54:42.591045 1581 raft_consensus.cc:697] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 LEADER]: Becoming Leader. State: Replica: f615c6cf70da4e8f8c4dac485b923dee, State: Running, Role: LEADER
I20260430 13:54:42.591593 1581 consensus_queue.cc:237] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:42.593364 1585 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f615c6cf70da4e8f8c4dac485b923dee. Latest consensus state: current_term: 6 leader_uuid: "f615c6cf70da4e8f8c4dac485b923dee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:42.593716 1585 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:42.594383 1586 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:54:42.596894 1586 catalog_manager.cc:679] Loaded metadata for table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0]
I20260430 13:54:42.597309 1586 catalog_manager.cc:679] Loaded metadata for table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02]
I20260430 13:54:42.597654 1586 catalog_manager.cc:679] Loaded metadata for table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56]
I20260430 13:54:42.597997 1586 catalog_manager.cc:679] Loaded metadata for table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a]
I20260430 13:54:42.599436 1586 tablet_loader.cc:96] loaded metadata for tablet 01949d383b2c44d583e43997ba847d49 (table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a])
I20260430 13:54:42.599789 1586 tablet_loader.cc:96] loaded metadata for tablet 349ee22fa537461ba9f48a3121f2c39c (table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0])
I20260430 13:54:42.600828 1586 tablet_loader.cc:96] loaded metadata for tablet 85be806c4e1b404d9d3e30cde7df0286 (table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56])
I20260430 13:54:42.601276 1586 tablet_loader.cc:96] loaded metadata for tablet a5821dad44c048548e5764a0dc87f967 (table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02])
I20260430 13:54:42.601595 1586 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:54:42.602730 1586 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:42.602815 1586 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:54:42.604164 1586 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:54:42.604986 1586 catalog_manager.cc:6055] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Loaded TSK: 0
I20260430 13:54:42.605857 1586 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:54:42.606660 1586 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:54:42.607438 1586 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:54:42.625294 804 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48570:
table { table_name: "default.Table_535de9acd8c747cba6ae09ea257f82c1" } new_table_name: "default.Table_0b74253e49c24199936c0fc4420538ed" modify_external_catalogs: true
I20260430 13:54:42.624722 807 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48570:
name: "default.Table_654fa8e50f25413abb27c763a47ecb4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.625182 803 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48570:
name: "default.Table_106dd7fc76304c5eadea2552b3735996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.625758 799 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48570:
name: "default.Table_9240d1b8bb1f4c99884d30bb311335d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.624698 806 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48570:
name: "default.Table_50797b6c7647446a904b4f32b529bdf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.625530 801 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48570:
table { table_name: "default.Table_646011822dc5429cb2f256da808338bc" } new_table_name: "default.Table_5928dba2f92a45df98b70003873a6ebe" modify_external_catalogs: true
I20260430 13:54:42.624835 805 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48570:
name: "default.Table_e16b7604faf34334980e2156642b296a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:42.626605 802 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48570:
table { table_name: "default.Table_fe99e22f9c214f8e9082c50a02935393" } new_table_name: "default.Table_da590a7148884972933145e283c0b67f" modify_external_catalogs: true
I20260430 13:54:42.626380 800 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48570:
table { table_name: "default.Table_38caf5dea0244ad6ba8033a82343784b" } new_table_name: "default.Table_4839c82734a34812b58582d1a7da4255" modify_external_catalogs: true
I20260430 13:54:42.626016 798 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48570:
name: "default.Table_5e7ce2408b714cc9b44c00d9d03919d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260430 13:54:42.638262 982 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 7 received events: NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_9240d1b8bb1f4c99884d30bb311335d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_9240d1b8bb1f4c99884d30bb311335d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9240d1b8bb1f4c99884d30bb311335d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_9240d1b8bb1f4c99884d30bb311335d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9240d1b8bb1f4c99884d30bb311335d1\",\"kudu.table_id\":\"f3ef536ae2384c829da92f20ac2d7c9c\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_5e7ce2408b714cc9b44c00d9d03919d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_5e7ce2408b714cc9b44c00d9d03919d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5e7ce2408b714cc9b44c00d9d03919d8\",\"kudu.table_id\":\"5aa844dc225b4ce09bb01e23e5321864\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f6029a6fdcd google::LogMessage::Fail() at ??:0
@ 0x7f6029a73b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6029a6f7cc google::LogMessage::Flush() at ??:0
@ 0x7f6029a6fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f6040524574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f6040522e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f604052aa9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f604052a919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7f602c808529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f602fdd66db start_thread at ??:0
@ 0x7f602892061f clone at ??:0
W20260430 13:54:42.908428 1145 connection.cc:570] client connection to 127.0.105.61:38801 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260430 13:54:42.909073 1408 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:38801)
W20260430 13:54:42.909744 1408 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:38801)
W20260430 13:54:42.910027 1004 connection.cc:570] client connection to 127.0.105.61:38801 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260430 13:54:43.248045 1600 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:43.248268 1600 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:43.250039 1600 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:43.251379 1506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:43.253283 1504 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:54:43.253545 1506 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:43.253705 1506 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:43.254065 1600 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:43.803153 1604 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:43.803355 1604 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:43.804154 1604 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:43.806457 1504 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:54:43.806774 1506 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:43.806910 1506 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:43.807193 1604 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:44.442519 420 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu pid 1471
I20260430 13:54:44.449143 1549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260430 13:54:44.562342 420 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu with pid 1471
I20260430 13:54:45.324548 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--webserver_interface=127.0.105.60
--webserver_port=43565
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260430 13:54:45.688117 1612 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:45.688490 1612 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:45.688612 1612 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:45.698271 1612 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:45.698378 1612 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:45.698443 1612 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:45.698506 1612 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:54:45.709389 1612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:45.711109 1612 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:45.713086 1612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:45.722069 1618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:45.722047 1617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:45.723212 1620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:45.724472 1612 server_base.cc:1061] running on GCE node
I20260430 13:54:45.725309 1612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:45.726958 1612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:45.728209 1612 hybrid_clock.cc:648] HybridClock initialized: now 1777557285728111 us; error 79 us; skew 500 ppm
I20260430 13:54:45.728652 1612 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:45.730895 1612 webserver.cc:492] Webserver started at http://127.0.105.60:43565/ using document root <none> and password file <none>
I20260430 13:54:45.731721 1612 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:45.731833 1612 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:45.738123 1612 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 13:54:45.741433 1626 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:45.743101 1612 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 13:54:45.743309 1612 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:45.744112 1612 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:45.772271 1612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:45.773257 1612 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:45.773561 1612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:45.792388 1612 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:41159
I20260430 13:54:45.792428 1677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:41159 every 8 connection(s)
I20260430 13:54:45.794054 1612 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
I20260430 13:54:45.801390 1678 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:54:45.803542 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 1612
I20260430 13:54:45.805243 1678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap starting.
I20260430 13:54:45.817809 1678 log.cc:826] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:45.829133 1678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260430 13:54:45.829762 1678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap complete.
I20260430 13:54:45.835237 1678 raft_consensus.cc:359] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:45.836208 1678 raft_consensus.cc:740] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Initialized, Role: FOLLOWER
I20260430 13:54:45.836864 1678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:45.837740 1683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:45.837980 1683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:45.838793 1678 sys_catalog.cc:565] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:54:45.851203 1697 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:45.851410 1697 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: loading cluster ID for follower catalog manager: success
I20260430 13:54:45.856077 1697 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: acquiring CA information for follower catalog manager: success
I20260430 13:54:45.859126 1697 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:54:45.860481 1612 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:54:45.864887 1612 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:45.904428 1254 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:45.906605 1643 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:35749; Asking this server to re-register.
I20260430 13:54:45.907331 1254 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:45.909080 1643 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:45.928679 1113 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:45.929911 1643 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:60577; Asking this server to re-register.
I20260430 13:54:45.930529 1113 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:45.931743 1643 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:54:46.072883 1395 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:46.074115 1643 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:60079; Asking this server to re-register.
I20260430 13:54:46.074771 1395 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:46.076196 1643 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:46.222576 1683 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:46.222864 1683 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:46.225217 1683 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:46.225883 1630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:46.227756 1630 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:46.228341 1628 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:54:46.228471 1628 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:46.228847 1683 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:46.883148 1703 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:46.883353 1703 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:46.884167 1703 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:46.886322 1628 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:54:46.886489 1630 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:46.886627 1630 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:46.886909 1703 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:47.924515 1707 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:47.924705 1707 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:47.925325 1707 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:47.927361 1630 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:47.927549 1628 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:54:47.927681 1628 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:47.927958 1707 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:48.158963 420 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu pid 1612
I20260430 13:54:48.172327 1673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260430 13:54:48.281912 420 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu with pid 1612
I20260430 13:54:49.044790 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--webserver_interface=127.0.105.60
--webserver_port=43565
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260430 13:54:49.383147 1715 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:49.383472 1715 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:49.383550 1715 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:49.393107 1715 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:49.393210 1715 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:49.393270 1715 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:49.393316 1715 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:54:49.403584 1715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:49.405229 1715 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:49.406986 1715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:49.415262 1720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:49.415249 1721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:49.416231 1723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:49.417596 1715 server_base.cc:1061] running on GCE node
I20260430 13:54:49.418464 1715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:49.420089 1715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:49.421311 1715 hybrid_clock.cc:648] HybridClock initialized: now 1777557289421249 us; error 58 us; skew 500 ppm
I20260430 13:54:49.421679 1715 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:49.424141 1715 webserver.cc:492] Webserver started at http://127.0.105.60:43565/ using document root <none> and password file <none>
I20260430 13:54:49.425263 1715 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:49.425369 1715 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:49.431114 1715 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260430 13:54:49.434494 1729 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:49.436679 1715 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260430 13:54:49.436900 1715 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:54:49.437932 1715 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:49.461776 1715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:49.462723 1715 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:49.463063 1715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:49.483634 1715 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:41159
I20260430 13:54:49.483712 1780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:41159 every 8 connection(s)
I20260430 13:54:49.485463 1715 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
I20260430 13:54:49.493026 1781 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:54:49.494138 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 1715
I20260430 13:54:49.496902 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap starting.
I20260430 13:54:49.506534 1781 log.cc:826] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:49.515703 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260430 13:54:49.516383 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap complete.
I20260430 13:54:49.521586 1781 raft_consensus.cc:359] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:49.522506 1781 raft_consensus.cc:740] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Initialized, Role: FOLLOWER
I20260430 13:54:49.523203 1781 consensus_queue.cc:260] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:49.524061 1785 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:49.524339 1785 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:49.525090 1781 sys_catalog.cc:565] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:54:49.534823 1715 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:54:49.540642 1715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:49.540762 1799 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:49.540874 1799 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: loading cluster ID for follower catalog manager: success
I20260430 13:54:49.544639 1799 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: acquiring CA information for follower catalog manager: success
I20260430 13:54:49.547313 1799 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:54:49.596197 1395 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:49.598727 1746 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:48547; Asking this server to re-register.
I20260430 13:54:49.599463 1395 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:49.601063 1746 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:49.928830 1254 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:49.930022 1746 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:57707; Asking this server to re-register.
I20260430 13:54:49.930574 1254 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:49.931593 1746 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:49.950505 1113 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:54:49.951434 1745 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:33607; Asking this server to re-register.
I20260430 13:54:49.951954 1113 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:49.953176 1745 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:54:50.119100 1806 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:50.119349 1806 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:50.121373 1806 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:50.122387 1731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:54:50.123862 1733 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:50.124168 1731 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:54:50.124300 1731 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:50.124632 1806 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:50.714377 1810 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:50.714562 1810 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:50.715286 1810 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:50.717478 1731 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:54:50.717679 1733 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:50.717818 1733 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:50.718078 1810 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:51.822474 1814 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:51.822664 1814 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:51.823344 1814 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:51.827240 1733 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:51.827428 1731 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:54:51.827569 1731 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:51.827912 1814 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 13:54:52.852408 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:35887
--webserver_interface=127.0.105.62
--webserver_port=38453
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260430 13:54:52.868625 1819 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:52.868818 1819 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:52.869513 1819 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:52.871781 1731 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:54:52.872251 1733 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:52.872375 1733 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:52.872648 1819 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 13:54:53.198302 1818 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:53.198609 1818 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:53.198709 1818 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:53.207509 1818 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:53.207639 1818 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:53.207746 1818 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:53.207813 1818 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:54:53.219352 1818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.105.62
--webserver_port=38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:53.220976 1818 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:53.222885 1818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:53.232235 1828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:53.232218 1827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:53.232179 1830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:53.233058 1818 server_base.cc:1061] running on GCE node
I20260430 13:54:53.233850 1818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:53.235425 1818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:53.236727 1818 hybrid_clock.cc:648] HybridClock initialized: now 1777557293236602 us; error 54 us; skew 500 ppm
I20260430 13:54:53.237123 1818 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:53.239517 1818 webserver.cc:492] Webserver started at http://127.0.105.62:38453/ using document root <none> and password file <none>
I20260430 13:54:53.240370 1818 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:53.240494 1818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:53.246516 1818 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 13:54:53.249887 1836 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:53.251551 1818 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 13:54:53.251757 1818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
uuid: "d392b5529f4245c397bd78166d0bc001"
format_stamp: "Formatted at 2026-04-30 13:54:36 on dist-test-slave-fmnx"
I20260430 13:54:53.252599 1818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:53.266763 1818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:53.267656 1818 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:53.267966 1818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:53.287571 1818 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.62:35887
I20260430 13:54:53.287575 1887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.62:35887 every 8 connection(s)
I20260430 13:54:53.289779 1818 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
I20260430 13:54:53.296697 1888 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:54:53.300122 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Bootstrap starting.
I20260430 13:54:53.300287 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 1818
I20260430 13:54:53.309758 1888 log.cc:826] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:53.323577 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 13:54:53.324182 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Bootstrap complete.
I20260430 13:54:53.329574 1888 raft_consensus.cc:359] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:53.330302 1888 raft_consensus.cc:740] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Initialized, Role: FOLLOWER
I20260430 13:54:53.330904 1888 consensus_queue.cc:260] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:53.331714 1892 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:53.332039 1892 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:53.332756 1888 sys_catalog.cc:565] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:54:53.345196 1906 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:53.345399 1906 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: loading cluster ID for follower catalog manager: success
I20260430 13:54:53.350937 1906 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: acquiring CA information for follower catalog manager: success
I20260430 13:54:53.354214 1906 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:54:53.362661 1818 master_runner.cc:186] Error getting master registration for 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111),
I20260430 13:54:53.368492 1818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:53.465853 1114 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:53.465840 1255 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:53.468346 1852 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:41037; Asking this server to re-register.
I20260430 13:54:53.468318 1853 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:46105; Asking this server to re-register.
I20260430 13:54:53.469041 1255 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:53.469157 1114 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:53.470709 1852 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:54:53.470829 1853 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:53.529206 1396 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:53.530478 1852 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:60023; Asking this server to re-register.
I20260430 13:54:53.531131 1396 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:53.532478 1852 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:53.863433 1911 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:53.863788 1911 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:53.865664 1911 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
W20260430 13:54:53.867106 1840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:53.868630 1840 leader_election.cc:336] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:53.872198 1756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:54:53.872679 1756 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d392b5529f4245c397bd78166d0bc001 in current term 6: Already voted for candidate f615c6cf70da4e8f8c4dac485b923dee in this term.
I20260430 13:54:53.873461 1838 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d392b5529f4245c397bd78166d0bc001; no voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3
I20260430 13:54:53.873921 1911 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 5 FOLLOWER]: Advancing to term 6
I20260430 13:54:53.877744 1911 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 13:54:54.551424 1715 master_runner.cc:186] Error getting master registration for 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111),
I20260430 13:54:54.551643 1715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:54.581633 1921 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:54.581838 1921 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:54.582571 1921 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:54.583261 1756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 7 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:54:54.583514 1756 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 6.
I20260430 13:54:54.584015 1838 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3; no voters:
I20260430 13:54:54.584602 1921 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 13:54:54.584739 1921 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 13:54:54.584765 1840 leader_election.cc:336] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:54.584851 1921 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 6 FOLLOWER]: Advancing to term 7
I20260430 13:54:54.587004 1921 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:54.587584 1921 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 7 election: Requested vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:54.588220 1756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 7 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:54:54.588413 1756 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 6 FOLLOWER]: Advancing to term 7
W20260430 13:54:54.588771 1840 leader_election.cc:336] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:54.592051 1756 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 7.
I20260430 13:54:54.592480 1838 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3; no voters: f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:54.592953 1921 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 13:54:54.593374 1921 raft_consensus.cc:697] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 LEADER]: Becoming Leader. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Running, Role: LEADER
I20260430 13:54:54.593871 1921 consensus_queue.cc:237] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:54.596514 1924 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d392b5529f4245c397bd78166d0bc001. Latest consensus state: current_term: 7 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:54.596822 1924 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:54.597438 1926 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:54:54.599783 1926 catalog_manager.cc:679] Loaded metadata for table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0]
I20260430 13:54:54.600255 1926 catalog_manager.cc:679] Loaded metadata for table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02]
I20260430 13:54:54.600592 1926 catalog_manager.cc:679] Loaded metadata for table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56]
I20260430 13:54:54.600914 1926 catalog_manager.cc:679] Loaded metadata for table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a]
I20260430 13:54:54.602372 1926 tablet_loader.cc:96] loaded metadata for tablet 01949d383b2c44d583e43997ba847d49 (table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a])
I20260430 13:54:54.603092 1926 tablet_loader.cc:96] loaded metadata for tablet 349ee22fa537461ba9f48a3121f2c39c (table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0])
I20260430 13:54:54.603384 1926 tablet_loader.cc:96] loaded metadata for tablet 85be806c4e1b404d9d3e30cde7df0286 (table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56])
I20260430 13:54:54.603690 1926 tablet_loader.cc:96] loaded metadata for tablet a5821dad44c048548e5764a0dc87f967 (table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02])
I20260430 13:54:54.604015 1926 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:54:54.604880 1926 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:54.604947 1926 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:54:54.606011 1926 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:54:54.606739 1926 catalog_manager.cc:6055] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Loaded TSK: 0
I20260430 13:54:54.607556 1926 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:54:54.608353 1926 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:54:54.609057 1926 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:54:54.703692 1853 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_c83031f7e5634413a1d01b8bce95a9d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:54.704311 1850 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_cc4a2daeb0de4cc182167aa213911136"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:54.704361 1849 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_5e7ce2408b714cc9b44c00d9d03919d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:54.704458 1848 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_9240d1b8bb1f4c99884d30bb311335d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:54.704483 1851 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_1d395d40fbbc4690af86be9b01485cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:54.703905 1852 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_b90a5e6d6e8b44748820ab72580fa9e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:54.705744 1847 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49860:
table { table_name: "default.Table_38caf5dea0244ad6ba8033a82343784b" } new_table_name: "default.Table_4839c82734a34812b58582d1a7da4255" modify_external_catalogs: true
I20260430 13:54:54.706943 1846 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49860:
table { table_name: "default.Table_646011822dc5429cb2f256da808338bc" } new_table_name: "default.Table_5928dba2f92a45df98b70003873a6ebe" modify_external_catalogs: true
I20260430 13:54:54.707093 1844 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_654fa8e50f25413abb27c763a47ecb4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:54.707747 1845 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49860:
name: "default.Table_50797b6c7647446a904b4f32b529bdf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260430 13:54:54.711690 1905 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 7 received events: NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_9240d1b8bb1f4c99884d30bb311335d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_9240d1b8bb1f4c99884d30bb311335d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9240d1b8bb1f4c99884d30bb311335d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_9240d1b8bb1f4c99884d30bb311335d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9240d1b8bb1f4c99884d30bb311335d1\",\"kudu.table_id\":\"f3ef536ae2384c829da92f20ac2d7c9c\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_5e7ce2408b714cc9b44c00d9d03919d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_5e7ce2408b714cc9b44c00d9d03919d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5e7ce2408b714cc9b44c00d9d03919d8\",\"kudu.table_id\":\"5aa844dc225b4ce09bb01e23e5321864\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f423570bdcd google::LogMessage::Fail() at ??:0
@ 0x7f423570fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f423570b7cc google::LogMessage::Flush() at ??:0
@ 0x7f423570ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f424c1c0574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f424c1bee21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f424c1c6a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f424c1c6919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7f42384a4529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f423ba726db start_thread at ??:0
@ 0x7f42345bc61f clone at ??:0
W20260430 13:54:54.904719 1406 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.62:35887)
W20260430 13:54:54.904983 1406 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.62:35887)
I20260430 13:54:56.315541 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:35887
--webserver_interface=127.0.105.62
--webserver_port=38453
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260430 13:54:56.650357 1942 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:54:56.650691 1942 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:54:56.650768 1942 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:54:56.659332 1942 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:54:56.659430 1942 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:54:56.659515 1942 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:54:56.659571 1942 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:54:56.670300 1942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.105.62
--webserver_port=38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:54:56.671948 1942 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:54:56.673727 1942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:54:56.682780 1948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:56.682752 1947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:54:56.683568 1950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:54:56.684316 1942 server_base.cc:1061] running on GCE node
I20260430 13:54:56.685218 1942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:54:56.686829 1942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:54:56.688040 1942 hybrid_clock.cc:648] HybridClock initialized: now 1777557296687986 us; error 47 us; skew 500 ppm
I20260430 13:54:56.688625 1942 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:54:56.690860 1942 webserver.cc:492] Webserver started at http://127.0.105.62:38453/ using document root <none> and password file <none>
I20260430 13:54:56.691700 1942 fs_manager.cc:362] Metadata directory not provided
I20260430 13:54:56.691816 1942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:54:56.697491 1942 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260430 13:54:56.700584 1956 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:54:56.702121 1942 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 13:54:56.702302 1942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
uuid: "d392b5529f4245c397bd78166d0bc001"
format_stamp: "Formatted at 2026-04-30 13:54:36 on dist-test-slave-fmnx"
I20260430 13:54:56.703085 1942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:54:56.720610 1942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:54:56.722038 1942 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:54:56.722708 1942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:54:56.743014 1942 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.62:35887
I20260430 13:54:56.743034 2007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.62:35887 every 8 connection(s)
I20260430 13:54:56.744755 1942 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-0/data/info.pb
I20260430 13:54:56.751332 2008 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:54:56.754030 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 1942
I20260430 13:54:56.754884 2008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Bootstrap starting.
I20260430 13:54:56.764667 2008 log.cc:826] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Log is configured to *not* fsync() on all Append() calls
I20260430 13:54:56.778010 2008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=10 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260430 13:54:56.778585 2008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Bootstrap complete.
I20260430 13:54:56.783830 2008 raft_consensus.cc:359] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:56.784816 2008 raft_consensus.cc:740] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Initialized, Role: FOLLOWER
I20260430 13:54:56.785414 2008 consensus_queue.cc:260] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 7.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:56.786280 2012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:56.786595 2012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:54:56.787302 2008 sys_catalog.cc:565] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:54:56.809914 2026 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:56.810117 2026 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: loading cluster ID for follower catalog manager: success
I20260430 13:54:56.814323 2026 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: acquiring CA information for follower catalog manager: success
I20260430 13:54:56.817247 2026 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:54:56.819084 1942 master_runner.cc:186] Error getting master registration for 127.0.105.61:38801: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111),
I20260430 13:54:56.823143 1942 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:54:56.988379 1114 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:56.988338 1255 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:56.990317 1972 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:37875; Asking this server to re-register.
I20260430 13:54:56.990345 1973 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:41899; Asking this server to re-register.
I20260430 13:54:56.990931 1114 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:56.991206 1255 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:56.992405 1973 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:54:56.992532 1972 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:54:57.049041 1396 heartbeater.cc:344] Connected to a master server at 127.0.105.62:35887
I20260430 13:54:57.050176 1972 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:46393; Asking this server to re-register.
I20260430 13:54:57.050729 1396 heartbeater.cc:461] Registering TS with master...
I20260430 13:54:57.051748 1972 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:54:57.175895 2031 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:57.176115 2031 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:57.176968 2031 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:57.178611 1733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20260430 13:54:57.180567 1733 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:57.184798 1983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d392b5529f4245c397bd78166d0bc001" is_pre_election: true
I20260430 13:54:57.185367 1983 raft_consensus.cc:2410] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f962b03e872d4cb990156ed62ea6afd3 for term 8 because replica has last-logged OpId of term: 7 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260430 13:54:57.186090 1731 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:57.186424 2031 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 13:54:57.396869 2035 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:57.397121 2035 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:57.398897 2035 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
W20260430 13:54:57.399816 1960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:54:57.401250 1960 leader_election.cc:336] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:57.405725 1756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 8 candidate_status { last_received { term: 7 index: 11 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:54:57.405958 1756 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 7.
I20260430 13:54:57.406394 1958 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3; no voters: f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:57.406785 2035 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 13:54:57.406919 2035 raft_consensus.cc:493] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 13:54:57.407015 2035 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 7 FOLLOWER]: Advancing to term 8
I20260430 13:54:57.411180 2035 raft_consensus.cc:515] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:57.411899 2035 leader_election.cc:290] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 8 election: Requested vote from peers f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:54:57.412896 1756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d392b5529f4245c397bd78166d0bc001" candidate_term: 8 candidate_status { last_received { term: 7 index: 11 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:54:57.413118 1756 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 7 FOLLOWER]: Advancing to term 8
W20260430 13:54:57.413820 1960 leader_election.cc:336] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:57.415109 1756 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392b5529f4245c397bd78166d0bc001 in term 8.
I20260430 13:54:57.415510 1958 leader_election.cc:304] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d392b5529f4245c397bd78166d0bc001, f962b03e872d4cb990156ed62ea6afd3; no voters: f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:57.415820 2035 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 13:54:57.416249 2035 raft_consensus.cc:697] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [term 8 LEADER]: Becoming Leader. State: Replica: d392b5529f4245c397bd78166d0bc001, State: Running, Role: LEADER
I20260430 13:54:57.416739 2035 consensus_queue.cc:237] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.11, Last appended by leader: 11, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:57.418922 2040 sys_catalog.cc:455] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d392b5529f4245c397bd78166d0bc001. Latest consensus state: current_term: 8 leader_uuid: "d392b5529f4245c397bd78166d0bc001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:54:57.419246 2040 sys_catalog.cc:458] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001 [sys.catalog]: This master's current role is: LEADER
I20260430 13:54:57.419946 2042 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:54:57.422417 2042 catalog_manager.cc:679] Loaded metadata for table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0]
I20260430 13:54:57.422847 2042 catalog_manager.cc:679] Loaded metadata for table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02]
I20260430 13:54:57.423187 2042 catalog_manager.cc:679] Loaded metadata for table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56]
I20260430 13:54:57.423519 2042 catalog_manager.cc:679] Loaded metadata for table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a]
I20260430 13:54:57.424959 2042 tablet_loader.cc:96] loaded metadata for tablet 01949d383b2c44d583e43997ba847d49 (table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a])
I20260430 13:54:57.425248 2042 tablet_loader.cc:96] loaded metadata for tablet 349ee22fa537461ba9f48a3121f2c39c (table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0])
I20260430 13:54:57.425508 2042 tablet_loader.cc:96] loaded metadata for tablet 85be806c4e1b404d9d3e30cde7df0286 (table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56])
I20260430 13:54:57.425760 2042 tablet_loader.cc:96] loaded metadata for tablet a5821dad44c048548e5764a0dc87f967 (table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02])
I20260430 13:54:57.426021 2042 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:54:57.426836 2042 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:54:57.426904 2042 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:54:57.428040 2042 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:54:57.428817 2042 catalog_manager.cc:6055] T 00000000000000000000000000000000 P d392b5529f4245c397bd78166d0bc001: Loaded TSK: 0
I20260430 13:54:57.429646 2042 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:54:57.430495 2042 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:54:57.431214 2042 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:54:57.489916 1972 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49844:
table { table_name: "default.Table_fe99e22f9c214f8e9082c50a02935393" } new_table_name: "default.Table_da590a7148884972933145e283c0b67f" modify_external_catalogs: true
I20260430 13:54:57.490224 1973 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49844:
name: "default.Table_106dd7fc76304c5eadea2552b3735996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:57.491317 1971 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49844:
table { table_name: "default.Table_535de9acd8c747cba6ae09ea257f82c1" } new_table_name: "default.Table_0b74253e49c24199936c0fc4420538ed" modify_external_catalogs: true
I20260430 13:54:57.491279 1964 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49844:
name: "default.Table_c83031f7e5634413a1d01b8bce95a9d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:57.491829 1970 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49844:
name: "default.Table_e16b7604faf34334980e2156642b296a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:57.492455 1969 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49844:
name: "default.Table_654fa8e50f25413abb27c763a47ecb4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:57.493000 1968 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49844:
name: "default.Table_50797b6c7647446a904b4f32b529bdf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:57.493923 1965 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49844:
name: "default.Table_b90a5e6d6e8b44748820ab72580fa9e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:54:57.494652 1967 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49844:
table { table_name: "default.Table_646011822dc5429cb2f256da808338bc" } new_table_name: "default.Table_5928dba2f92a45df98b70003873a6ebe" modify_external_catalogs: true
I20260430 13:54:57.494707 1966 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49844:
table { table_name: "default.Table_38caf5dea0244ad6ba8033a82343784b" } new_table_name: "default.Table_4839c82734a34812b58582d1a7da4255" modify_external_catalogs: true
F20260430 13:54:57.498893 2025 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 7 received events: NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_9240d1b8bb1f4c99884d30bb311335d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_9240d1b8bb1f4c99884d30bb311335d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9240d1b8bb1f4c99884d30bb311335d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_9240d1b8bb1f4c99884d30bb311335d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9240d1b8bb1f4c99884d30bb311335d1\",\"kudu.table_id\":\"f3ef536ae2384c829da92f20ac2d7c9c\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_5e7ce2408b714cc9b44c00d9d03919d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_5e7ce2408b714cc9b44c00d9d03919d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5e7ce2408b714cc9b44c00d9d03919d8\",\"kudu.table_id\":\"5aa844dc225b4ce09bb01e23e5321864\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f152f585dcd google::LogMessage::Fail() at ??:0
@ 0x7f152f589b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f152f5857cc google::LogMessage::Flush() at ??:0
@ 0x7f152f585a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f154603a574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f1546038e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f1546040a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f1546040919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7f153231e529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f15358ec6db start_thread at ??:0
@ 0x7f152e43661f clone at ??:0
W20260430 13:54:57.684991 1284 connection.cc:570] client connection to 127.0.105.62:35887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260430 13:54:57.685909 1406 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.62:35887)
W20260430 13:54:57.686657 1406 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.62:35887)
I20260430 13:54:58.566080 2058 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:54:58.566349 2058 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:54:58.566988 2058 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:54:58.569404 1731 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:54:58.569662 1733 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:54:58.569768 1733 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:54:58.570016 2058 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 13:54:59.217029 420 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu pid 1715
I20260430 13:54:59.230237 1776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260430 13:54:59.341542 420 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu with pid 1715
I20260430 13:55:00.152614 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--webserver_interface=127.0.105.60
--webserver_port=43565
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260430 13:55:00.495812 2066 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:55:00.496217 2066 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:55:00.496296 2066 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:55:00.505100 2066 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:55:00.505200 2066 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:55:00.505259 2066 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:55:00.505314 2066 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:55:00.515697 2066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:55:00.517406 2066 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:55:00.519210 2066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:55:00.528172 2074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:00.528131 2071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:00.528084 2072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:55:00.528818 2066 server_base.cc:1061] running on GCE node
I20260430 13:55:00.529590 2066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:55:00.531123 2066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:55:00.532373 2066 hybrid_clock.cc:648] HybridClock initialized: now 1777557300532296 us; error 71 us; skew 500 ppm
I20260430 13:55:00.532783 2066 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:55:00.535189 2066 webserver.cc:492] Webserver started at http://127.0.105.60:43565/ using document root <none> and password file <none>
I20260430 13:55:00.535998 2066 fs_manager.cc:362] Metadata directory not provided
I20260430 13:55:00.536116 2066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:55:00.542934 2066 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260430 13:55:00.546169 2080 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:55:00.547649 2066 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 13:55:00.547811 2066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:55:00.548543 2066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:55:00.589358 2066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:55:00.590231 2066 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:55:00.590507 2066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:55:00.610082 2131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:41159 every 8 connection(s)
I20260430 13:55:00.610075 2066 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:41159
I20260430 13:55:00.612280 2066 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
I20260430 13:55:00.619542 2132 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:55:00.621893 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 2066
I20260430 13:55:00.623178 2132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap starting.
I20260430 13:55:00.631946 2132 log.cc:826] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Log is configured to *not* fsync() on all Append() calls
I20260430 13:55:00.640720 2132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260430 13:55:00.641755 2132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap complete.
I20260430 13:55:00.647631 1395 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:55:00.649029 2132 raft_consensus.cc:359] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:00.650136 2132 raft_consensus.cc:740] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Initialized, Role: FOLLOWER
I20260430 13:55:00.650787 2132 consensus_queue.cc:260] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:00.651571 2136 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:55:00.651815 2136 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:55:00.652750 2132 sys_catalog.cc:565] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:55:00.669668 2150 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:55:00.669792 2150 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: loading cluster ID for follower catalog manager: success
I20260430 13:55:00.675094 2150 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: acquiring CA information for follower catalog manager: success
I20260430 13:55:00.677990 2150 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:55:00.673008 2066 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:00.678841 2066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:00.975128 1254 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:55:00.977070 2097 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:56573; Asking this server to re-register.
I20260430 13:55:00.977872 1254 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:00.979903 2097 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:55:00.996371 1113 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:55:00.997644 2097 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:54089; Asking this server to re-register.
I20260430 13:55:00.998267 1113 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:00.999585 2097 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:55:01.097390 2136 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:01.097661 2136 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:01.099632 2136 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:01.100354 2082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:01.101850 2082 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:01.102164 2084 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:55:01.102281 2084 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:01.102633 2136 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 13:55:01.151163 2097 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:41953; Asking this server to re-register.
I20260430 13:55:01.151855 1395 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:01.153137 2097 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:55:01.682973 2157 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:01.683183 2157 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:01.683951 2157 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:01.686183 2084 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:01.686430 2082 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:01.686538 2082 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:01.686792 2157 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 13:55:02.632653 2161 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:02.632835 2161 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:02.633514 2161 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:02.635546 2082 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:02.635715 2084 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:55:02.635826 2084 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:02.636109 2161 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 13:55:03.816115 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:38801
--webserver_interface=127.0.105.61
--webserver_port=43365
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260430 13:55:03.878054 2166 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:03.878249 2166 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:03.878960 2166 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:03.881368 2082 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:03.881536 2084 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:55:03.881685 2084 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:03.882133 2166 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260430 13:55:04.157873 2165 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:55:04.158203 2165 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:55:04.158303 2165 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:55:04.167548 2165 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:55:04.167655 2165 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:55:04.167734 2165 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:55:04.167793 2165 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:55:04.178645 2165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=43365
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:55:04.180349 2165 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:55:04.182240 2165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:55:04.190536 2174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:04.190546 2175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:04.191093 2177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:55:04.191975 2165 server_base.cc:1061] running on GCE node
I20260430 13:55:04.192764 2165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:55:04.194249 2165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:55:04.195452 2165 hybrid_clock.cc:648] HybridClock initialized: now 1777557304195412 us; error 31 us; skew 500 ppm
I20260430 13:55:04.195808 2165 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:55:04.198094 2165 webserver.cc:492] Webserver started at http://127.0.105.61:43365/ using document root <none> and password file <none>
I20260430 13:55:04.198951 2165 fs_manager.cc:362] Metadata directory not provided
I20260430 13:55:04.199070 2165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:55:04.204996 2165 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260430 13:55:04.208622 2183 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:55:04.210237 2165 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260430 13:55:04.210394 2165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
uuid: "f615c6cf70da4e8f8c4dac485b923dee"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:55:04.211086 2165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:55:04.242522 2165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:55:04.243461 2165 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:55:04.243721 2165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:55:04.268934 2165 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:38801
I20260430 13:55:04.269040 2234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:38801 every 8 connection(s)
I20260430 13:55:04.270673 2165 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
I20260430 13:55:04.275187 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 2165
I20260430 13:55:04.278065 2235 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:55:04.281946 2235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Bootstrap starting.
I20260430 13:55:04.291011 2235 log.cc:826] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Log is configured to *not* fsync() on all Append() calls
I20260430 13:55:04.303467 2235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=10 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260430 13:55:04.304141 2235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Bootstrap complete.
I20260430 13:55:04.309427 2235 raft_consensus.cc:359] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:04.310382 2235 raft_consensus.cc:740] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: f615c6cf70da4e8f8c4dac485b923dee, State: Initialized, Role: FOLLOWER
I20260430 13:55:04.311021 2235 consensus_queue.cc:260] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 6.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:04.311853 2239 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:55:04.312171 2239 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:55:04.313611 2235 sys_catalog.cc:565] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:55:04.325178 2253 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:55:04.325387 2253 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: loading cluster ID for follower catalog manager: success
I20260430 13:55:04.329751 2253 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: acquiring CA information for follower catalog manager: success
I20260430 13:55:04.332695 2253 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:55:04.334100 2165 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:04.339748 2165 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:04.470270 1115 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:55:04.470477 1256 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:55:04.472672 2200 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:55821; Asking this server to re-register.
I20260430 13:55:04.472692 2199 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:36603; Asking this server to re-register.
I20260430 13:55:04.473541 1115 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:04.473634 1256 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:04.475801 2200 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:55:04.475728 2199 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:55:04.505976 1397 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:55:04.507095 2199 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:36833; Asking this server to re-register.
I20260430 13:55:04.507687 1397 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:04.508940 2199 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:55:04.516736 2258 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:04.516938 2258 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:04.517738 2258 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:04.522112 2082 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:04.526464 2210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee" is_pre_election: true
I20260430 13:55:04.527099 2210 raft_consensus.cc:2410] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f962b03e872d4cb990156ed62ea6afd3 for term 9 because replica has last-logged OpId of term: 6 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260430 13:55:04.528036 2084 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:04.528352 2258 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 13:55:04.827963 2262 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:04.828251 2262 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:04.830123 2262 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
W20260430 13:55:04.831105 2185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:04.832867 2185 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:04.837428 2107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:55:04.837846 2107 raft_consensus.cc:2368] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f615c6cf70da4e8f8c4dac485b923dee for earlier term 7. Current term is 8.
I20260430 13:55:04.838590 2185 leader_election.cc:400] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 7 pre-election: Vote denied by peer f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f615c6cf70da4e8f8c4dac485b923dee for earlier term 7. Current term is 8.
I20260430 13:55:04.838670 2185 leader_election.cc:403] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260430 13:55:04.838924 2262 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 6 FOLLOWER]: Advancing to term 8
I20260430 13:55:04.842680 2262 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f615c6cf70da4e8f8c4dac485b923dee for earlier term 7. Current term is 8.
I20260430 13:55:05.423931 2269 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:05.424157 2269 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:05.424817 2269 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:55:05.426047 2107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 9 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:55:05.426295 2107 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f615c6cf70da4e8f8c4dac485b923dee in term 8.
I20260430 13:55:05.426960 2185 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters:
W20260430 13:55:05.427284 2185 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:05.427289 2269 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 13:55:05.427590 2269 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 13:55:05.427681 2269 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 8 FOLLOWER]: Advancing to term 9
I20260430 13:55:05.429733 2269 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:05.430346 2269 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 9 election: Requested vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:55:05.431207 2107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 9 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:55:05.431396 2107 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 8 FOLLOWER]: Advancing to term 9
W20260430 13:55:05.431577 2185 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:05.434919 2107 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f615c6cf70da4e8f8c4dac485b923dee in term 9.
I20260430 13:55:05.435374 2185 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001
I20260430 13:55:05.435708 2269 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Leader election won for term 9
I20260430 13:55:05.436162 2269 raft_consensus.cc:697] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 LEADER]: Becoming Leader. State: Replica: f615c6cf70da4e8f8c4dac485b923dee, State: Running, Role: LEADER
I20260430 13:55:05.436619 2269 consensus_queue.cc:237] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 6.11, Last appended by leader: 11, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:05.439199 2272 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f615c6cf70da4e8f8c4dac485b923dee. Latest consensus state: current_term: 9 leader_uuid: "f615c6cf70da4e8f8c4dac485b923dee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:55:05.439433 2272 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: LEADER
I20260430 13:55:05.440145 2274 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:55:05.442574 2274 catalog_manager.cc:679] Loaded metadata for table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0]
I20260430 13:55:05.443010 2274 catalog_manager.cc:679] Loaded metadata for table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02]
I20260430 13:55:05.443354 2274 catalog_manager.cc:679] Loaded metadata for table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56]
I20260430 13:55:05.443730 2274 catalog_manager.cc:679] Loaded metadata for table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a]
I20260430 13:55:05.445175 2274 tablet_loader.cc:96] loaded metadata for tablet 01949d383b2c44d583e43997ba847d49 (table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a])
I20260430 13:55:05.445453 2274 tablet_loader.cc:96] loaded metadata for tablet 349ee22fa537461ba9f48a3121f2c39c (table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0])
I20260430 13:55:05.445681 2274 tablet_loader.cc:96] loaded metadata for tablet 85be806c4e1b404d9d3e30cde7df0286 (table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56])
I20260430 13:55:05.445904 2274 tablet_loader.cc:96] loaded metadata for tablet a5821dad44c048548e5764a0dc87f967 (table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02])
I20260430 13:55:05.446130 2274 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:55:05.447367 2274 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:55:05.447464 2274 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:55:05.448894 2274 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:55:05.449637 2274 catalog_manager.cc:6055] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Loaded TSK: 0
I20260430 13:55:05.450436 2274 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:55:05.451200 2274 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:55:05.451987 2274 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:55:05.486275 2200 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49002:
name: "default.Table_5e7ce2408b714cc9b44c00d9d03919d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:05.486470 2198 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49002:
name: "default.Table_cc4a2daeb0de4cc182167aa213911136"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:05.486555 2196 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49002:
name: "default.Table_c83031f7e5634413a1d01b8bce95a9d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:05.487120 2193 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49002:
table { table_name: "default.Table_646011822dc5429cb2f256da808338bc" } new_table_name: "default.Table_5928dba2f92a45df98b70003873a6ebe" modify_external_catalogs: true
I20260430 13:55:05.487361 2197 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49002:
name: "default.Table_1d395d40fbbc4690af86be9b01485cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:05.487601 2192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49002:
name: "default.Table_106dd7fc76304c5eadea2552b3735996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:05.488145 2194 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49002:
table { table_name: "default.Table_38caf5dea0244ad6ba8033a82343784b" } new_table_name: "default.Table_4839c82734a34812b58582d1a7da4255" modify_external_catalogs: true
I20260430 13:55:05.486316 2199 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49002:
name: "default.Table_9240d1b8bb1f4c99884d30bb311335d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:05.488588 2195 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49002:
name: "default.Table_b90a5e6d6e8b44748820ab72580fa9e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:05.488794 2191 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49002:
table { table_name: "default.Table_fe99e22f9c214f8e9082c50a02935393" } new_table_name: "default.Table_da590a7148884972933145e283c0b67f" modify_external_catalogs: true
F20260430 13:55:05.494069 2252 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 7 received events: NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_9240d1b8bb1f4c99884d30bb311335d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_9240d1b8bb1f4c99884d30bb311335d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9240d1b8bb1f4c99884d30bb311335d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_9240d1b8bb1f4c99884d30bb311335d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9240d1b8bb1f4c99884d30bb311335d1\",\"kudu.table_id\":\"f3ef536ae2384c829da92f20ac2d7c9c\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_5e7ce2408b714cc9b44c00d9d03919d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_5e7ce2408b714cc9b44c00d9d03919d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5e7ce2408b714cc9b44c00d9d03919d8\",\"kudu.table_id\":\"5aa844dc225b4ce09bb01e23e5321864\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fced40f4dcd google::LogMessage::Fail() at ??:0
@ 0x7fced40f8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fced40f47cc google::LogMessage::Flush() at ??:0
@ 0x7fced40f4a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fceeaba9574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fceeaba7e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fceeabafa9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fceeabaf919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fced6e8d529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fceda45b6db start_thread at ??:0
@ 0x7fced2fa561f clone at ??:0
I20260430 13:55:05.680569 2066 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:05.680819 2066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260430 13:55:05.692581 1408 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:38801)
W20260430 13:55:05.693156 1408 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:38801)
I20260430 13:55:05.980933 1256 heartbeater.cc:499] Master 127.0.105.61:38801 was elected leader, sending a full tablet report...
I20260430 13:55:05.980966 1115 heartbeater.cc:499] Master 127.0.105.61:38801 was elected leader, sending a full tablet report...
I20260430 13:55:06.713950 420 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu pid 2066
I20260430 13:55:06.721272 2127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260430 13:55:06.828698 420 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu with pid 2066
I20260430 13:55:07.777168 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--webserver_interface=127.0.105.60
--webserver_port=43565
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260430 13:55:08.122896 2295 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:55:08.123242 2295 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:55:08.123346 2295 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:55:08.132117 2295 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:55:08.132234 2295 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:55:08.132313 2295 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:55:08.132370 2295 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:55:08.143638 2295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:55:08.145458 2295 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:55:08.147540 2295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:55:08.156632 2300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:08.156654 2301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:08.157377 2303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:55:08.158913 2295 server_base.cc:1061] running on GCE node
I20260430 13:55:08.159817 2295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:55:08.161506 2295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:55:08.162767 2295 hybrid_clock.cc:648] HybridClock initialized: now 1777557308162700 us; error 52 us; skew 500 ppm
I20260430 13:55:08.163224 2295 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:55:08.165997 2295 webserver.cc:492] Webserver started at http://127.0.105.60:43565/ using document root <none> and password file <none>
I20260430 13:55:08.166795 2295 fs_manager.cc:362] Metadata directory not provided
I20260430 13:55:08.166896 2295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:55:08.172691 2295 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260430 13:55:08.176242 2309 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:55:08.177968 2295 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260430 13:55:08.178165 2295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
uuid: "f962b03e872d4cb990156ed62ea6afd3"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:55:08.179037 2295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:55:08.209951 2295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:55:08.210912 2295 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:55:08.211249 2295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:55:08.232393 2295 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:41159
I20260430 13:55:08.232400 2360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:41159 every 8 connection(s)
I20260430 13:55:08.234057 2295 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-2/data/info.pb
I20260430 13:55:08.236198 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 2295
I20260430 13:55:08.242879 2361 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:55:08.247496 2361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap starting.
I20260430 13:55:08.256340 2361 log.cc:826] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Log is configured to *not* fsync() on all Append() calls
I20260430 13:55:08.267663 2361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260430 13:55:08.268585 2361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: Bootstrap complete.
I20260430 13:55:08.275169 2361 raft_consensus.cc:359] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:08.276187 2361 raft_consensus.cc:740] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: f962b03e872d4cb990156ed62ea6afd3, State: Initialized, Role: FOLLOWER
I20260430 13:55:08.276850 2361 consensus_queue.cc:260] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:08.277691 2365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:55:08.277957 2365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:55:08.278791 2361 sys_catalog.cc:565] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:55:08.296324 2379 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:55:08.299233 2379 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: loading cluster ID for follower catalog manager: success
I20260430 13:55:08.300650 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:08.306896 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:08.308447 2379 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: acquiring CA information for follower catalog manager: success
I20260430 13:55:08.311290 2379 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:55:08.512495 1254 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:55:08.514640 2325 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:46419; Asking this server to re-register.
I20260430 13:55:08.515352 1254 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:08.517004 2325 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:55:08.530726 1113 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:55:08.532130 2325 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:52399; Asking this server to re-register.
I20260430 13:55:08.532810 1113 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:08.534232 2325 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:55:08.685151 1395 heartbeater.cc:344] Connected to a master server at 127.0.105.60:41159
I20260430 13:55:08.686159 2325 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:50981; Asking this server to re-register.
I20260430 13:55:08.686623 1395 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:08.687736 2325 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:55:08.815578 2383 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:08.815868 2383 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:08.818063 2383 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:08.818969 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:08.820286 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:08.820528 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:08.820689 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:08.821022 2383 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 13:55:09.408910 2387 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:09.409089 2387 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:09.409781 2387 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:09.411839 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:09.412058 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:09.412196 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:09.412482 2387 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 13:55:10.200351 2391 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:10.200562 2391 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:10.201377 2391 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:10.204285 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:10.204510 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:10.204735 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:10.205139 2391 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 13:55:11.148649 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
/tmp/dist-test-taskZgxaqK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:38801
--webserver_interface=127.0.105.61
--webserver_port=43365
--builtin_ntp_servers=127.0.105.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260430 13:55:11.506211 2395 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 13:55:11.506551 2395 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 13:55:11.506656 2395 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 13:55:11.515686 2395 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 13:55:11.515801 2395 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 13:55:11.515981 2395 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 13:55:11.516064 2395 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 13:55:11.527004 2395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40107
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=43365
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type FASTDEBUG
built by None at 30 Apr 2026 13:43:24 UTC on bdcb31816ec0
build id 11678
ASAN enabled
I20260430 13:55:11.528802 2395 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 13:55:11.530647 2395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 13:55:11.539711 2400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:11.539902 2403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 13:55:11.539705 2401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 13:55:11.540673 2395 server_base.cc:1061] running on GCE node
I20260430 13:55:11.541519 2395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 13:55:11.543048 2395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 13:55:11.544312 2395 hybrid_clock.cc:648] HybridClock initialized: now 1777557311544241 us; error 73 us; skew 500 ppm
I20260430 13:55:11.544708 2395 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 13:55:11.547104 2395 webserver.cc:492] Webserver started at http://127.0.105.61:43365/ using document root <none> and password file <none>
I20260430 13:55:11.547921 2395 fs_manager.cc:362] Metadata directory not provided
I20260430 13:55:11.548027 2395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 13:55:11.554826 2395 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260430 13:55:11.558210 2409 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 13:55:11.559782 2395 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 13:55:11.560002 2395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
uuid: "f615c6cf70da4e8f8c4dac485b923dee"
format_stamp: "Formatted at 2026-04-30 13:54:37 on dist-test-slave-fmnx"
I20260430 13:55:11.560824 2395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 13:55:11.581343 2395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 13:55:11.582357 2395 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 13:55:11.582629 2395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 13:55:11.602129 2395 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:38801
I20260430 13:55:11.602167 2460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:38801 every 8 connection(s)
I20260430 13:55:11.603713 2395 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/master-1/data/info.pb
I20260430 13:55:11.608651 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskZgxaqK/build/asan/bin/kudu as pid 2395
I20260430 13:55:11.611027 2461 sys_catalog.cc:263] Verifying existing consensus state
I20260430 13:55:11.616127 2461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Bootstrap starting.
I20260430 13:55:11.627317 2461 log.cc:826] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Log is configured to *not* fsync() on all Append() calls
I20260430 13:55:11.640182 2461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=10 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260430 13:55:11.640777 2461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Bootstrap complete.
I20260430 13:55:11.646173 2461 raft_consensus.cc:359] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:11.647106 2461 raft_consensus.cc:740] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: f615c6cf70da4e8f8c4dac485b923dee, State: Initialized, Role: FOLLOWER
I20260430 13:55:11.647706 2461 consensus_queue.cc:260] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 9.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:11.648566 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:55:11.648816 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: FOLLOWER
I20260430 13:55:11.649564 2461 sys_catalog.cc:565] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: configured and running, proceeding with master startup.
I20260430 13:55:11.664547 2480 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:55:11.664674 2480 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: loading cluster ID for follower catalog manager: success
I20260430 13:55:11.668526 2480 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: acquiring CA information for follower catalog manager: success
I20260430 13:55:11.671264 2480 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 13:55:11.672498 2395 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:11.676540 2395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:11.711109 1397 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:55:11.713533 2426 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f512a4ecfc764c1abbdfc4a37dc20d4f" instance_seqno: 1777557280401530) as {username='slave'} at 127.0.105.3:59001; Asking this server to re-register.
I20260430 13:55:11.714246 1397 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:11.715914 2426 ts_manager.cc:194] Registered new tserver with Master: f512a4ecfc764c1abbdfc4a37dc20d4f (127.0.105.3:46429)
I20260430 13:55:11.812453 2484 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:11.812664 2484 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:11.813463 2484 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:11.816279 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:11.821343 2436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f962b03e872d4cb990156ed62ea6afd3" candidate_term: 10 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f615c6cf70da4e8f8c4dac485b923dee" is_pre_election: true
I20260430 13:55:11.822088 2436 raft_consensus.cc:2410] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f962b03e872d4cb990156ed62ea6afd3 for term 10 because replica has last-logged OpId of term: 9 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260430 13:55:11.823124 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:11.823503 2484 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 13:55:12.003414 1115 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:55:12.003679 1115 heartbeater.cc:499] Master 127.0.105.61:38801 was elected leader, sending a full tablet report...
I20260430 13:55:12.004202 1256 heartbeater.cc:344] Connected to a master server at 127.0.105.61:38801
I20260430 13:55:12.004391 1256 heartbeater.cc:499] Master 127.0.105.61:38801 was elected leader, sending a full tablet report...
I20260430 13:55:12.004709 2426 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "72423ef5b9b54d4690d5050d879590d7" instance_seqno: 1777557279168290) as {username='slave'} at 127.0.105.1:42805; Asking this server to re-register.
I20260430 13:55:12.005227 1115 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:12.005357 2425 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "fde79c28b7c94c3f9466dcc8516fbad5" instance_seqno: 1777557279705655) as {username='slave'} at 127.0.105.2:36483; Asking this server to re-register.
I20260430 13:55:12.005887 1256 heartbeater.cc:461] Registering TS with master...
I20260430 13:55:12.007041 2425 ts_manager.cc:194] Registered new tserver with Master: 72423ef5b9b54d4690d5050d879590d7 (127.0.105.1:44493)
I20260430 13:55:12.007102 2426 ts_manager.cc:194] Registered new tserver with Master: fde79c28b7c94c3f9466dcc8516fbad5 (127.0.105.2:35227)
I20260430 13:55:12.153944 2488 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:12.154295 2488 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:12.156152 2488 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
W20260430 13:55:12.157016 2411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:12.158814 2411 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:12.163671 2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 10 candidate_status { last_received { term: 9 index: 12 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3" is_pre_election: true
I20260430 13:55:12.164145 2336 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f615c6cf70da4e8f8c4dac485b923dee in term 9.
I20260430 13:55:12.164812 2411 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001
I20260430 13:55:12.165191 2488 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 13:55:12.165354 2488 raft_consensus.cc:493] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 13:55:12.165462 2488 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 9 FOLLOWER]: Advancing to term 10
I20260430 13:55:12.170358 2488 raft_consensus.cc:515] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:12.171068 2488 leader_election.cc:290] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 10 election: Requested vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f962b03e872d4cb990156ed62ea6afd3 (127.0.105.60:41159)
I20260430 13:55:12.171860 2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f615c6cf70da4e8f8c4dac485b923dee" candidate_term: 10 candidate_status { last_received { term: 9 index: 12 } } ignore_live_leader: false dest_uuid: "f962b03e872d4cb990156ed62ea6afd3"
I20260430 13:55:12.172106 2336 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 9 FOLLOWER]: Advancing to term 10
W20260430 13:55:12.172264 2411 leader_election.cc:336] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:12.175801 2336 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f615c6cf70da4e8f8c4dac485b923dee in term 10.
I20260430 13:55:12.176524 2411 leader_election.cc:304] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f615c6cf70da4e8f8c4dac485b923dee, f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001
I20260430 13:55:12.176891 2488 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 10 FOLLOWER]: Leader election won for term 10
I20260430 13:55:12.177309 2488 raft_consensus.cc:697] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [term 10 LEADER]: Becoming Leader. State: Replica: f615c6cf70da4e8f8c4dac485b923dee, State: Running, Role: LEADER
I20260430 13:55:12.177785 2488 consensus_queue.cc:237] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.12, Last appended by leader: 12, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:12.180770 2492 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f615c6cf70da4e8f8c4dac485b923dee. Latest consensus state: current_term: 10 leader_uuid: "f615c6cf70da4e8f8c4dac485b923dee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } } }
I20260430 13:55:12.181154 2492 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee [sys.catalog]: This master's current role is: LEADER
I20260430 13:55:12.181963 2494 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 13:55:12.184505 2494 catalog_manager.cc:679] Loaded metadata for table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0]
I20260430 13:55:12.184952 2494 catalog_manager.cc:679] Loaded metadata for table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02]
I20260430 13:55:12.185364 2494 catalog_manager.cc:679] Loaded metadata for table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56]
I20260430 13:55:12.185757 2494 catalog_manager.cc:679] Loaded metadata for table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a]
I20260430 13:55:12.187282 2494 tablet_loader.cc:96] loaded metadata for tablet 01949d383b2c44d583e43997ba847d49 (table default.table_535de9acd8c747cba6ae09ea257f82c1 [id=bae9523e823c4703ac8463cb06496d1a])
I20260430 13:55:12.187582 2494 tablet_loader.cc:96] loaded metadata for tablet 349ee22fa537461ba9f48a3121f2c39c (table default.table_38caf5dea0244ad6ba8033a82343784b [id=1e3228304b9349cf86cc688efee611f0])
I20260430 13:55:12.187809 2494 tablet_loader.cc:96] loaded metadata for tablet 85be806c4e1b404d9d3e30cde7df0286 (table default.table_fe99e22f9c214f8e9082c50a02935393 [id=6c15f9d20dca40b4a95132cc17279a56])
I20260430 13:55:12.188063 2494 tablet_loader.cc:96] loaded metadata for tablet a5821dad44c048548e5764a0dc87f967 (table default.table_646011822dc5429cb2f256da808338bc [id=635b8a8981244f7d84fc2e58a88b9a02])
I20260430 13:55:12.188304 2494 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 13:55:12.189177 2494 catalog_manager.cc:1269] Loaded cluster ID: feb56f29d1e6411c8ef9ce8404345b01
I20260430 13:55:12.189253 2494 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 13:55:12.190397 2494 catalog_manager.cc:1514] Loading token signing keys...
I20260430 13:55:12.191148 2494 catalog_manager.cc:6055] T 00000000000000000000000000000000 P f615c6cf70da4e8f8c4dac485b923dee: Loaded TSK: 0
I20260430 13:55:12.192067 2494 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 13:55:12.192821 2494 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260430 13:55:12.193523 2494 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260430 13:55:12.300796 2425 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50838:
name: "default.Table_654fa8e50f25413abb27c763a47ecb4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:12.300895 2426 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50838:
name: "default.Table_50797b6c7647446a904b4f32b529bdf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:12.301167 2423 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50838:
table { table_name: "default.Table_535de9acd8c747cba6ae09ea257f82c1" } new_table_name: "default.Table_0b74253e49c24199936c0fc4420538ed" modify_external_catalogs: true
I20260430 13:55:12.301766 2424 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50838:
name: "default.Table_e16b7604faf34334980e2156642b296a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:12.302364 2420 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50838:
table { table_name: "default.Table_646011822dc5429cb2f256da808338bc" } new_table_name: "default.Table_5928dba2f92a45df98b70003873a6ebe" modify_external_catalogs: true
I20260430 13:55:12.302338 2422 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50838:
table { table_name: "default.Table_fe99e22f9c214f8e9082c50a02935393" } new_table_name: "default.Table_da590a7148884972933145e283c0b67f" modify_external_catalogs: true
I20260430 13:55:12.302839 2417 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50838:
name: "default.Table_9240d1b8bb1f4c99884d30bb311335d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:12.303049 2419 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50838:
table { table_name: "default.Table_38caf5dea0244ad6ba8033a82343784b" } new_table_name: "default.Table_4839c82734a34812b58582d1a7da4255" modify_external_catalogs: true
I20260430 13:55:12.302870 2421 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50838:
name: "default.Table_106dd7fc76304c5eadea2552b3735996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 13:55:12.302412 2418 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50838:
name: "default.Table_5e7ce2408b714cc9b44c00d9d03919d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260430 13:55:12.308506 2479 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 7 received events: NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_9240d1b8bb1f4c99884d30bb311335d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_9240d1b8bb1f4c99884d30bb311335d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9240d1b8bb1f4c99884d30bb311335d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_9240d1b8bb1f4c99884d30bb311335d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9240d1b8bb1f4c99884d30bb311335d1\",\"kudu.table_id\":\"f3ef536ae2384c829da92f20ac2d7c9c\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777557280, eventType=CREATE_TABLE, dbName=default, tableName=table_5e7ce2408b714cc9b44c00d9d03919d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_5e7ce2408b714cc9b44c00d9d03919d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777557280},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskZgxaqK/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777557259398929-420-0/minicluster-data/warehouse/table_5e7ce2408b714cc9b44c00d9d03919d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5e7ce2408b714cc9b44c00d9d03919d8\",\"kudu.table_id\":\"5aa844dc225b4ce09bb01e23e5321864\",\"transient_lastDdlTime\":\"1777557280\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"feb56f29d1e6411c8ef9ce8404345b01\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777557280,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fa3e5f3cdcd google::LogMessage::Fail() at ??:0
@ 0x7fa3e5f40b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa3e5f3c7cc google::LogMessage::Flush() at ??:0
@ 0x7fa3e5f3ca11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fa3fc9f1574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fa3fc9efe21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fa3fc9f7a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fa3fc9f7919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fa3e8cd5529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa3ec2a36db start_thread at ??:0
@ 0x7fa3e4ded61f clone at ??:0
W20260430 13:55:12.501212 1408 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:38801)
W20260430 13:55:12.501960 1408 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:38801)
I20260430 13:55:12.719628 1397 heartbeater.cc:499] Master 127.0.105.61:38801 was elected leader, sending a full tablet report...
I20260430 13:55:13.308458 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:13.308645 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:14.169854 420 master-stress-test.cc:577] Killing test thread 1/25
I20260430 13:55:15.549024 2509 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:15.549234 2509 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:15.549866 2509 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:15.551014 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260430 13:55:15.552376 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:15.552527 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:15.552672 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:15.552933 2509 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:55:16.408813 2513 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:16.409065 2513 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:16.409679 2513 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:16.413684 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:16.413857 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:16.413987 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:16.414225 2513 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:55:18.310323 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:18.310528 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:21.827149 2540 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:21.827358 2540 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:21.828099 2540 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:21.829115 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260430 13:55:21.830349 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:21.830600 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:21.830703 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:21.830938 2540 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:55:23.312157 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:23.312350 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:26.331128 2563 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:26.331357 2563 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:26.332005 2563 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:26.334997 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:26.335160 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:26.335294 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:26.335582 2563 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:55:28.313913 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:28.314229 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:32.775839 2594 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:32.776054 2594 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:32.776682 2594 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:32.777621 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260430 13:55:32.778731 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:32.778889 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:55:32.778988 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:32.779322 2594 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:55:33.316076 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:33.316344 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:34.725888 2605 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:34.726081 2605 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:34.726788 2605 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:34.728942 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:34.729126 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:34.729228 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:34.729466 2605 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:55:38.318051 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:38.318260 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260430 13:55:41.304881 1397 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (57 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:55:41.381206 2636 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:41.381433 2636 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:41.382157 2636 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:41.383203 2313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260430 13:55:41.384443 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:41.384605 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:55:41.384809 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:41.385136 2636 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260430 13:55:41.580331 1256 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (58 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:55:41.580363 1115 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (58 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:55:43.319931 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:43.320171 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:48.321851 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:48.322058 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:51.219806 2680 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:55:51.220052 2680 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:55:51.220710 2680 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:55:51.224486 2313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:55:51.225880 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:55:51.226097 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:55:51.226217 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:55:51.226531 2680 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:55:53.328997 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:53.329206 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:55:58.330827 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:55:58.331058 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:03.332779 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:03.332978 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:03.836927 2737 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:56:03.837127 2737 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:56:03.837903 2737 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:56:03.838898 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:56:03.840286 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:56:03.840500 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:56:03.840649 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:56:03.841001 2737 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:56:08.334570 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:08.334767 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:13.336339 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:13.336534 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:18.338405 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:18.338619 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:20.942530 2814 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:56:20.942708 2814 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:56:20.943718 2814 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:56:20.944669 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:56:20.946046 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:56:20.946269 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:56:20.946372 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:56:20.946602 2814 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:56:23.340373 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:23.340580 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:27.406186 2843 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:56:27.406383 2843 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:56:27.407060 2843 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:56:27.408123 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:56:27.409430 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:56:27.409590 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:56:27.409725 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:56:27.409992 2843 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:56:28.342016 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:28.342247 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:33.343477 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:33.343654 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:38.345350 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:38.345546 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:39.392261 2898 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:56:39.392457 2898 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:56:39.393196 2898 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:56:39.394395 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:56:39.395526 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:56:39.395664 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:56:39.395793 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:56:39.396183 2898 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260430 13:56:41.484449 1396 heartbeater.cc:646] Failed to heartbeat to 127.0.105.62:35887 (207 consecutive failures): Network error: Failed to ping master at 127.0.105.62:35887: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:56:41.723384 1256 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (178 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:56:41.723472 1115 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (178 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:56:42.275187 2914 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:56:42.275391 2914 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:56:42.276207 2914 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:56:42.278414 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:56:42.278623 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:56:42.278731 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:56:42.278997 2914 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:56:43.347172 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:43.347385 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:48.349126 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:48.349385 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:53.351091 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:53.351295 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:56:54.211818 2968 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:56:54.212059 2968 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:56:54.212795 2968 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:56:54.213858 2313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260430 13:56:54.214885 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:56:54.215050 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:56:54.215186 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:56:54.215392 2968 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:56:58.353787 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:56:58.353996 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:03.355473 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:03.355757 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:08.357242 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:08.357421 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:08.392449 2315 maintenance_manager.cc:419] P f962b03e872d4cb990156ed62ea6afd3: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033365
I20260430 13:57:08.433204 2314 maintenance_manager.cc:643] P f962b03e872d4cb990156ed62ea6afd3: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.013s sys 0.012s Metrics: {"bytes_written":6027,"cfile_init":1,"dirs.queue_time_us":390,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1103,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":3,"thread_start_us":468,"threads_started":2,"wal-append.queue_time_us":321}
I20260430 13:57:08.434374 2315 maintenance_manager.cc:419] P f962b03e872d4cb990156ed62ea6afd3: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260430 13:57:08.435050 2314 maintenance_manager.cc:643] P f962b03e872d4cb990156ed62ea6afd3: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20260430 13:57:12.141263 3049 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:57:12.141486 3049 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:57:12.142419 3049 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:57:12.143657 2313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:57:12.145032 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:57:12.145192 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:57:12.145341 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:57:12.145565 3049 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:57:13.358870 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:13.359066 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:18.360713 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:18.360971 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:23.362509 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:23.362699 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:28.364284 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:28.364480 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:28.965211 3167 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:57:28.965425 3167 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:57:28.966321 3167 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:57:28.967289 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:57:28.968523 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:57:28.968706 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:57:28.968813 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:57:28.969097 3167 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:57:33.366051 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:33.366247 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:38.367790 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:38.368047 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260430 13:57:41.663153 1396 heartbeater.cc:646] Failed to heartbeat to 127.0.105.62:35887 (327 consecutive failures): Network error: Failed to ping master at 127.0.105.62:35887: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:57:41.870236 1115 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (298 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:57:41.870224 1256 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (298 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:57:43.369588 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:43.369772 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:46.804463 3312 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:57:46.804673 3312 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:57:46.805539 3312 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:57:46.806536 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:57:46.807857 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:57:46.808092 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:57:46.808213 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:57:46.808466 3312 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:57:48.371292 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:48.371488 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:53.373229 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:53.373481 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:57:58.375032 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:57:58.375249 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:03.376901 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:03.377105 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:04.774585 3454 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:58:04.774793 3454 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:58:04.775717 3454 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:58:04.776858 2313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:58:04.778246 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:58:04.778466 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:58:04.778585 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:58:04.778837 3454 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:58:08.378782 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:08.378998 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:13.380722 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:13.380970 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:18.382772 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:18.382975 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:20.749663 3547 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:58:20.749866 3547 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:58:20.750782 3547 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:58:20.751904 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:58:20.753032 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:58:20.753196 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:58:20.753311 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:58:20.753582 3547 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:58:23.384626 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:23.384845 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:26.105638 3590 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:58:26.105844 3590 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:58:26.106554 3590 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:58:26.107404 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:58:26.108887 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:58:26.109052 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:58:26.109179 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:58:26.109433 3590 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:58:28.386308 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:28.386513 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:33.388118 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:33.388319 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:38.389889 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:38.390166 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:40.805572 3703 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:58:40.805783 3703 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:58:40.806414 3703 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:58:40.807368 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:58:40.808560 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:58:40.808707 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:58:40.808825 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:58:40.809023 3703 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260430 13:58:41.839951 1397 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (417 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:58:42.019083 1115 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (418 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:58:42.019101 1256 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:38801 (418 consecutive failures): Network error: Failed to ping master at 127.0.105.61:38801: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:58:43.391968 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:43.392189 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:48.393908 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:48.394184 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:53.395674 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:53.395923 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:58:56.415637 3820 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:58:56.415961 3820 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:58:56.416865 3820 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:58:56.417755 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:58:56.419118 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:58:56.419274 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:58:56.419406 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:58:56.419658 3820 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:58:58.397668 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:58:58.397886 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:03.399474 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:03.399696 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:03.726511 3875 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:59:03.726751 3875 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:59:03.727479 3875 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:59:03.728206 2313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:59:03.729686 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:59:03.729843 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:59:03.729977 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:59:03.730221 3875 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:59:08.401189 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:08.401410 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:13.403005 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:13.403189 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:16.179541 3966 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:59:16.179770 3966 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:59:16.180593 3966 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:59:16.181614 2313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260430 13:59:16.183245 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:59:16.183411 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:59:16.183557 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:59:16.183846 3966 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:59:18.405058 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:18.405272 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:20.224180 3999 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:59:20.224378 3999 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:59:20.225046 3999 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:59:20.227129 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
W20260430 13:59:20.227334 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
I20260430 13:59:20.227494 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:59:20.227733 3999 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:59:23.406752 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:23.406956 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:28.408602 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:28.408813 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:32.634677 4088 raft_consensus.cc:493] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 13:59:32.634899 4088 raft_consensus.cc:515] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d392b5529f4245c397bd78166d0bc001" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 35887 } } peers { permanent_uuid: "f615c6cf70da4e8f8c4dac485b923dee" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 38801 } } peers { permanent_uuid: "f962b03e872d4cb990156ed62ea6afd3" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 41159 } }
I20260430 13:59:32.635668 4088 leader_election.cc:290] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887), f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801)
W20260430 13:59:32.636740 2311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260430 13:59:32.637913 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f615c6cf70da4e8f8c4dac485b923dee (127.0.105.61:38801): Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:59:32.638068 2311 leader_election.cc:336] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d392b5529f4245c397bd78166d0bc001 (127.0.105.62:35887): Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111)
I20260430 13:59:32.638201 2311 leader_election.cc:304] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f962b03e872d4cb990156ed62ea6afd3; no voters: d392b5529f4245c397bd78166d0bc001, f615c6cf70da4e8f8c4dac485b923dee
I20260430 13:59:32.638486 4088 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f962b03e872d4cb990156ed62ea6afd3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 13:59:33.410419 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:33.410634 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:38.412472 2295 master_runner.cc:186] Error getting master registration for 127.0.105.62:35887: Network error: Client connection negotiation failed: client connection to 127.0.105.62:35887: connect: Connection refused (error 111),
I20260430 13:59:38.412701 2295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260430 13:59:38.669015 2315 maintenance_manager.cc:419] P f962b03e872d4cb990156ed62ea6afd3: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260430 13:59:38.673838 2314 maintenance_manager.cc:643] P f962b03e872d4cb990156ed62ea6afd3: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.004s sys 0.001s Metrics: {"dirs.queue_time_us":381,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":97,"thread_start_us":484,"threads_started":2,"wal-append.queue_time_us":324}
W20260430 13:59:40.642508 1406 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
W20260430 13:59:40.642784 1406 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
F20260430 13:59:40.654983 1427 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b90a5e6d6e8b44748820ab72580fa9e8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
*** Check failure stack trace: ***
W20260430 13:59:40.655180 1408 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:38801)
*** Aborted at 1777557580 (unix time) try "date -d @1777557580" if you are using GNU date ***
W20260430 13:59:40.656795 1408 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:38801)
F20260430 13:59:40.658958 1429 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d395d40fbbc4690af86be9b01485cf4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260430 13:59:40.658958 1429 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d395d40fbbc4690af86be9b01485cf4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.658959 1424 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4a2daeb0de4cc182167aa213911136 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260430 13:59:40.658958 1429 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d395d40fbbc4690af86be9b01485cf4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.658959 1424 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4a2daeb0de4cc182167aa213911136 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.660250 1428 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c83031f7e5634413a1d01b8bce95a9d3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260430 13:59:40.658958 1429 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d395d40fbbc4690af86be9b01485cf4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.658959 1424 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4a2daeb0de4cc182167aa213911136 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.660250 1428 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c83031f7e5634413a1d01b8bce95a9d3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.660840 1426 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5e7ce2408b714cc9b44c00d9d03919d8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260430 13:59:40.658958 1429 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d395d40fbbc4690af86be9b01485cf4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.658959 1424 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4a2daeb0de4cc182167aa213911136 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.660250 1428 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c83031f7e5634413a1d01b8bce95a9d3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.660840 1426 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5e7ce2408b714cc9b44c00d9d03919d8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)F20260430 13:59:40.662642 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9240d1b8bb1f4c99884d30bb311335d1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:35887,127.0.105.61:38801,127.0.105.60:41159, num_attempts: 728) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.61:38801: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000001a4) received by PID 420 (TID 0x7fb4508ad700) from PID 420; stack trace: ***
@ 0x7fb466c15980 (unknown) at ??:0
@ 0x7fb45fa05e87 gsignal at ??:0
@ 0x7fb45fa077f1 abort at ??:0
@ 0x7fb46273fdcd google::LogMessage::Fail() at ??:0
@ 0x7fb462743b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb46273f7cc google::LogMessage::Flush() at ??:0
@ 0x7fb462740f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b582 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fb46008d6df (unknown) at ??:0
@ 0x7fb466c0a6db start_thread at ??:0
@ 0x7fb45fae861f clone at ??:0