Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-05-02T07:57:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-02T07:57:00Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXQ2lRx/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXQ2lRx/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/security.properties
2026-05-02 07:57:07: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXQ2lRx/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXQ2lRx/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260502 07:57:15.430070 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:43729
--webserver_interface=127.0.105.62
--webserver_port=0
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 07:57:15.750075 704 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:15.750334 704 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:15.750388 704 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:15.758671 704 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:15.758747 704 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:15.758801 704 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:15.758845 704 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:15.768159 704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:15.769604 704 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:15.771241 704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:15.779284 709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:15.779314 712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:15.779819 710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:15.780069 704 server_base.cc:1061] running on GCE node
I20260502 07:57:15.780936 704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:15.782138 704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:15.783356 704 hybrid_clock.cc:648] HybridClock initialized: now 1777708635783275 us; error 71 us; skew 500 ppm
I20260502 07:57:15.783735 704 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:15.785868 704 webserver.cc:492] Webserver started at http://127.0.105.62:42221/ using document root <none> and password file <none>
I20260502 07:57:15.786618 704 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:15.786747 704 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:15.787127 704 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 07:57:15.789283 704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/instance:
uuid: "364807e94fd04d41bfa5976f76f187a6"
format_stamp: "Formatted at 2026-05-02 07:57:15 on dist-test-slave-sj7c"
I20260502 07:57:15.789996 704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal/instance:
uuid: "364807e94fd04d41bfa5976f76f187a6"
format_stamp: "Formatted at 2026-05-02 07:57:15 on dist-test-slave-sj7c"
I20260502 07:57:15.794716 704 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260502 07:57:15.797850 718 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:15.798998 704 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260502 07:57:15.799199 704 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
uuid: "364807e94fd04d41bfa5976f76f187a6"
format_stamp: "Formatted at 2026-05-02 07:57:15 on dist-test-slave-sj7c"
I20260502 07:57:15.799400 704 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:15.833631 704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:15.834520 704 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:15.834796 704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:15.856000 704 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.62:43729
I20260502 07:57:15.856020 769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.62:43729 every 8 connection(s)
I20260502 07:57:15.857544 704 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/info.pb
I20260502 07:57:15.861480 770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 07:57:15.866556 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 704
I20260502 07:57:15.866868 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal/instance
I20260502 07:57:15.867944 770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260502 07:57:15.870185 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--webserver_interface=127.0.105.61
--webserver_port=0
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 07:57:15.885056 770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
W20260502 07:57:15.886061 722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:15.887930 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:15.937335 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } attempt: 1
W20260502 07:57:15.940008 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:16.008389 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } attempt: 2
W20260502 07:57:16.010257 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:16.101608 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } attempt: 3
W20260502 07:57:16.103487 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:16.185966 773 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:16.186260 773 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:16.186425 773 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:16.194772 773 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:16.194854 773 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:16.194908 773 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:16.194944 773 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:16.204393 773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:16.205986 773 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:16.208299 773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:16.216686 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:16.216614 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:16.216583 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:16.217367 773 server_base.cc:1061] running on GCE node
I20260502 07:57:16.218073 773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:16.219395 773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:16.220651 773 hybrid_clock.cc:648] HybridClock initialized: now 1777708636220608 us; error 43 us; skew 500 ppm
I20260502 07:57:16.220992 773 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:16.222956 773 webserver.cc:492] Webserver started at http://127.0.105.61:40445/ using document root <none> and password file <none>
I20260502 07:57:16.223760 773 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:16.223889 773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:16.224211 773 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 07:57:16.228586 773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/instance:
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:16.229803 773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal/instance:
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:16.235100 773 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260502 07:57:16.238257 788 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:16.239722 773 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260502 07:57:16.239919 773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:16.240127 773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:16.246721 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } attempt: 4
W20260502 07:57:16.248885 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:16.290347 773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:16.291702 773 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:16.291982 773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:16.312752 773 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:33363
I20260502 07:57:16.312777 839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:33363 every 8 connection(s)
I20260502 07:57:16.314088 773 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
I20260502 07:57:16.318111 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 07:57:16.318372 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 773
I20260502 07:57:16.318576 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal/instance
I20260502 07:57:16.319962 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:35545
--webserver_interface=127.0.105.60
--webserver_port=0
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 07:57:16.323786 840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260502 07:57:16.336797 840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
I20260502 07:57:16.346082 840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } has no permanent_uuid. Determining permanent_uuid...
W20260502 07:57:16.347030 792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
W20260502 07:57:16.348579 840 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:16.397985 840 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 1
W20260502 07:57:16.399809 840 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:16.468016 840 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 2
W20260502 07:57:16.469878 840 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:16.548198 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } attempt: 5
I20260502 07:57:16.554471 770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } has no permanent_uuid. Determining permanent_uuid...
W20260502 07:57:16.556593 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:16.561076 840 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 3
W20260502 07:57:16.562673 840 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:16.607873 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 1
W20260502 07:57:16.609678 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
W20260502 07:57:16.651018 841 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:16.651351 841 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:16.651419 841 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:16.660006 841 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:16.660091 841 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:16.660154 841 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:16.660202 841 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:16.670147 841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:35545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:16.671600 841 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:16.673187 841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260502 07:57:16.677942 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 2
W20260502 07:57:16.680114 770 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
W20260502 07:57:16.681463 850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:16.681538 849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:16.683830 852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:16.684027 841 server_base.cc:1061] running on GCE node
I20260502 07:57:16.684749 841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:16.685985 841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:16.687198 841 hybrid_clock.cc:648] HybridClock initialized: now 1777708636687150 us; error 59 us; skew 500 ppm
I20260502 07:57:16.687556 841 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:16.689474 841 webserver.cc:492] Webserver started at http://127.0.105.60:44821/ using document root <none> and password file <none>
I20260502 07:57:16.690137 841 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:16.690258 841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:16.690577 841 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 07:57:16.692655 841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/instance:
uuid: "0b49a409ffb84415ad9b53c0a862c470"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:16.693327 841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal/instance:
uuid: "0b49a409ffb84415ad9b53c0a862c470"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:16.698081 841 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260502 07:57:16.701547 858 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:16.702703 841 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260502 07:57:16.702875 841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
uuid: "0b49a409ffb84415ad9b53c0a862c470"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:16.703075 841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:16.705853 840 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 4
W20260502 07:57:16.707684 840 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:16.735608 841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:16.736485 841 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:16.736755 841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:16.756909 841 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:35545
I20260502 07:57:16.756928 909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:35545 every 8 connection(s)
I20260502 07:57:16.758347 841 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
I20260502 07:57:16.762601 910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 07:57:16.767328 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 841
I20260502 07:57:16.767534 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal/instance
I20260502 07:57:16.768316 910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260502 07:57:16.782601 910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
I20260502 07:57:16.786314 770 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 3
I20260502 07:57:16.790274 910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } has no permanent_uuid. Determining permanent_uuid...
I20260502 07:57:16.802443 770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Bootstrap starting.
I20260502 07:57:16.806233 770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Neither blocks nor log segments found. Creating new log.
I20260502 07:57:16.807989 770 log.cc:826] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:16.809558 910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Bootstrap starting.
I20260502 07:57:16.813081 910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Neither blocks nor log segments found. Creating new log.
I20260502 07:57:16.814011 910 log.cc:826] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:16.814429 770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: No bootstrap required, opened a new log
I20260502 07:57:16.816464 910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: No bootstrap required, opened a new log
I20260502 07:57:16.821512 770 raft_consensus.cc:359] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:16.821820 770 raft_consensus.cc:385] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 07:57:16.821931 770 raft_consensus.cc:740] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364807e94fd04d41bfa5976f76f187a6, State: Initialized, Role: FOLLOWER
I20260502 07:57:16.822522 770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:16.823119 910 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:16.823429 910 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 07:57:16.823541 910 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b49a409ffb84415ad9b53c0a862c470, State: Initialized, Role: FOLLOWER
I20260502 07:57:16.823900 920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:16.824148 920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:16.824079 910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:16.824816 921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:16.825059 770 sys_catalog.cc:565] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:16.825065 921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:16.825902 910 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: configured and running, proceeding with master startup.
W20260502 07:57:16.838340 724 tablet.cc:2404] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260502 07:57:16.840705 945 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260502 07:57:16.840917 945 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260502 07:57:16.841830 704 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: OK, instance_id { permanent_uuid: "51250627cebd4f9abefd40c111865403" instance_seqno: 1777708636307553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260502 07:57:16.850108 704 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:16.880888 841 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: OK, instance_id { permanent_uuid: "51250627cebd4f9abefd40c111865403" instance_seqno: 1777708636307553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260502 07:57:16.893016 841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 07:57:16.895725 952 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260502 07:57:16.895906 952 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260502 07:57:16.910684 920 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:16.910915 920 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:16.912536 920 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:16.914453 885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:16.915011 885 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 364807e94fd04d41bfa5976f76f187a6 in term 0.
I20260502 07:57:16.917464 722 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 364807e94fd04d41bfa5976f76f187a6; no voters:
I20260502 07:57:16.918270 920 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260502 07:57:16.918404 920 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:16.918490 920 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 0 FOLLOWER]: Advancing to term 1
I20260502 07:57:16.921241 920 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:16.923460 885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470"
I20260502 07:57:16.923491 920 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 1 election: Requested vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:16.923635 885 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 0 FOLLOWER]: Advancing to term 1
I20260502 07:57:16.926648 885 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 364807e94fd04d41bfa5976f76f187a6 in term 1.
I20260502 07:57:16.927218 722 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 364807e94fd04d41bfa5976f76f187a6; no voters:
I20260502 07:57:16.927532 920 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260502 07:57:16.928058 920 raft_consensus.cc:697] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 1 LEADER]: Becoming Leader. State: Replica: 364807e94fd04d41bfa5976f76f187a6, State: Running, Role: LEADER
I20260502 07:57:16.928517 920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:16.940244 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364807e94fd04d41bfa5976f76f187a6. Latest consensus state: current_term: 1 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:16.940527 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:16.951642 957 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:16.953728 957 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:16.973251 885 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 1 FOLLOWER]: Refusing update from remote peer 364807e94fd04d41bfa5976f76f187a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260502 07:57:16.974149 955 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260502 07:57:16.988727 864 tablet.cc:2404] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260502 07:57:16.998586 921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364807e94fd04d41bfa5976f76f187a6. Latest consensus state: current_term: 1 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:16.998819 921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:17.002357 920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:17.002575 920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:17.002980 959 mvcc.cc:204] Tried to move back new op lower bound from 7281494577033662464 to 7281494576864153600. Current Snapshot: MvccSnapshot[applied={T|T < 7281494577033662464}]
I20260502 07:57:17.004472 957 catalog_manager.cc:1357] Generated new cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:17.004593 957 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:17.007122 840 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } attempt: 5
I20260502 07:57:17.008056 921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:17.008251 921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:17.025100 840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap starting.
I20260502 07:57:17.033860 840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Neither blocks nor log segments found. Creating new log.
I20260502 07:57:17.037227 840 log.cc:826] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:17.037806 957 catalog_manager.cc:1380] Generated new certificate authority record
I20260502 07:57:17.041718 957 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:17.048830 840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: No bootstrap required, opened a new log
I20260502 07:57:17.062991 840 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:17.063925 840 raft_consensus.cc:385] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 07:57:17.064241 840 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Initialized, Role: FOLLOWER
I20260502 07:57:17.065109 957 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Generated new TSK 0
I20260502 07:57:17.066237 957 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:17.066468 840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:17.067713 957 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:17.069026 957 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:17.069742 964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:17.070017 964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:17.071669 840 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:17.072043 813 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 0 FOLLOWER]: Advancing to term 1
I20260502 07:57:17.075145 814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403"
I20260502 07:57:17.071688 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403" is_pre_election: true
I20260502 07:57:17.079484 813 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 1 FOLLOWER]: Refusing update from remote peer 364807e94fd04d41bfa5976f76f187a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260502 07:57:17.080910 955 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260502 07:57:17.090687 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.105.1:0
--local_ip_for_outbound_sockets=127.0.105.1
--webserver_interface=127.0.105.1
--webserver_port=0
--tserver_master_addrs=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260502 07:57:17.145766 964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364807e94fd04d41bfa5976f76f187a6. Latest consensus state: current_term: 1 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:17.145700 976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:17.145952 964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:17.146008 976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:17.189894 985 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:17.189998 985 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: loading cluster ID for follower catalog manager: success
I20260502 07:57:17.193739 985 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: acquiring CA information for follower catalog manager: success
I20260502 07:57:17.195946 985 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:17.565907 987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 364807e94fd04d41bfa5976f76f187a6)
I20260502 07:57:17.566131 987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:17.567708 987 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
I20260502 07:57:17.568305 745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6" is_pre_election: true
I20260502 07:57:17.568472 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403" is_pre_election: true
I20260502 07:57:17.569200 859 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:17.569600 987 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260502 07:57:17.590049 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:17.590284 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:17.591099 976 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 364807e94fd04d41bfa5976f76f187a6)
I20260502 07:57:17.591310 976 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:17.592784 976 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:17.593933 745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6" is_pre_election: true
I20260502 07:57:17.595446 884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:17.595669 884 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 1.
I20260502 07:57:17.596155 792 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:17.597384 976 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260502 07:57:17.597560 976 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 364807e94fd04d41bfa5976f76f187a6)
I20260502 07:57:17.597681 976 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 1 FOLLOWER]: Advancing to term 2
I20260502 07:57:17.602422 976 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:17.603480 745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6"
I20260502 07:57:17.603545 884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470"
I20260502 07:57:17.603706 884 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 1 FOLLOWER]: Advancing to term 2
I20260502 07:57:17.604112 976 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 2 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:17.606920 884 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 2.
I20260502 07:57:17.607370 792 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:17.607702 976 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 2 FOLLOWER]: Leader election won for term 2
I20260502 07:57:17.607918 976 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 2 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:17.608455 976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:17.610087 964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 2 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:17.610265 964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:17.611037 994 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:17.613350 884 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Rejecting Update request from peer 364807e94fd04d41bfa5976f76f187a6 for earlier term 1. Current term is 2. Ops: []
I20260502 07:57:17.613989 994 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:17.613924 988 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260502 07:57:17.614187 988 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 1 LEADER]: Stepping down as leader of term 1
I20260502 07:57:17.614276 988 raft_consensus.cc:740] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 364807e94fd04d41bfa5976f76f187a6, State: Running, Role: LEADER
I20260502 07:57:17.614486 988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:17.614925 988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 1 FOLLOWER]: Advancing to term 2
I20260502 07:57:17.614991 994 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:17.615084 994 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
W20260502 07:57:17.615195 968 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:17.615499 968 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:17.615610 968 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260502 07:57:17.616611 994 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:17.618271 994 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:17.619367 994 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:17.620455 994 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:17.621600 994 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
W20260502 07:57:17.626554 968 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:17.626778 968 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.105.1
I20260502 07:57:17.637008 968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.105.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.105.1
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:17.638566 968 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:17.640329 968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:17.650027 996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:17.650069 997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:17.652856 999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:17.653005 968 server_base.cc:1061] running on GCE node
I20260502 07:57:17.653517 968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:17.654305 968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:17.655506 968 hybrid_clock.cc:648] HybridClock initialized: now 1777708637655456 us; error 35 us; skew 500 ppm
I20260502 07:57:17.655802 968 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:17.657910 968 webserver.cc:492] Webserver started at http://127.0.105.1:40519/ using document root <none> and password file <none>
I20260502 07:57:17.658576 968 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:17.658676 968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:17.659008 968 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 07:57:17.661304 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data/instance:
uuid: "3216d6df438a4041a7d61fb516f73ed3"
format_stamp: "Formatted at 2026-05-02 07:57:17 on dist-test-slave-sj7c"
I20260502 07:57:17.662042 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/wal/instance:
uuid: "3216d6df438a4041a7d61fb516f73ed3"
format_stamp: "Formatted at 2026-05-02 07:57:17 on dist-test-slave-sj7c"
I20260502 07:57:17.667011 968 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260502 07:57:17.670161 1005 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:17.671490 968 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260502 07:57:17.671646 968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/wal
uuid: "3216d6df438a4041a7d61fb516f73ed3"
format_stamp: "Formatted at 2026-05-02 07:57:17 on dist-test-slave-sj7c"
I20260502 07:57:17.671851 968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:17.693251 968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:17.694072 968 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:17.694316 968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:17.695266 968 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260502 07:57:17.696879 968 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260502 07:57:17.697002 968 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:17.697116 968 ts_tablet_manager.cc:616] Registered 0 tablets
I20260502 07:57:17.697196 968 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:17.734578 968 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.1:41957
I20260502 07:57:17.734674 1117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.1:41957 every 8 connection(s)
I20260502 07:57:17.735957 968 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/data/info.pb
I20260502 07:57:17.743175 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 968
I20260502 07:57:17.743490 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-0/wal/instance
I20260502 07:57:17.751940 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.105.2:0
--local_ip_for_outbound_sockets=127.0.105.2
--webserver_interface=127.0.105.2
--webserver_port=0
--tserver_master_addrs=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260502 07:57:17.780393 1118 heartbeater.cc:344] Connected to a master server at 127.0.105.62:43729
I20260502 07:57:17.780822 1118 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:17.781811 1118 heartbeater.cc:507] Master 127.0.105.62:43729 requested a full tablet report, sending...
I20260502 07:57:17.783830 1119 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:17.783947 1119 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:17.784128 735 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:17.784385 1119 heartbeater.cc:507] Master 127.0.105.60:35545 requested a full tablet report, sending...
I20260502 07:57:17.785768 875 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:17.793155 1120 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:17.793256 1120 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:17.793684 1120 heartbeater.cc:507] Master 127.0.105.61:33363 requested a full tablet report, sending...
I20260502 07:57:17.795081 805 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:17.797673 805 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.105.1:42155
I20260502 07:57:17.843392 945 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:17.897588 952 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:17.897679 952 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: loading cluster ID for follower catalog manager: success
I20260502 07:57:17.900207 952 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: acquiring CA information for follower catalog manager: success
I20260502 07:57:17.901814 952 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:18.038560 884 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Refusing update from remote peer 51250627cebd4f9abefd40c111865403: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260502 07:57:18.039341 964 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260502 07:57:18.040951 987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 2 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:18.041222 987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:18.042615 964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:18.042847 964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:18.043421 745 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 2 FOLLOWER]: Refusing update from remote peer 51250627cebd4f9abefd40c111865403: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260502 07:57:18.044159 964 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260502 07:57:18.044200 987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:18.044363 987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:18.046753 988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:18.047034 988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:18.046916 989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 2 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:18.047204 989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: FOLLOWER
W20260502 07:57:18.094640 1126 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:18.094918 1126 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:18.094995 1126 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:18.103945 1126 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:18.104095 1126 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.105.2
I20260502 07:57:18.113929 1126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.105.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.105.2
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:18.115417 1126 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:18.117063 1126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:18.126585 1136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:18.126638 1137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:18.127354 1139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:18.127969 1126 server_base.cc:1061] running on GCE node
I20260502 07:57:18.128492 1126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:18.129402 1126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:18.130622 1126 hybrid_clock.cc:648] HybridClock initialized: now 1777708638130569 us; error 68 us; skew 500 ppm
I20260502 07:57:18.130885 1126 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:18.133347 1126 webserver.cc:492] Webserver started at http://127.0.105.2:38539/ using document root <none> and password file <none>
I20260502 07:57:18.134150 1126 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:18.134282 1126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:18.134624 1126 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 07:57:18.136931 1126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data/instance:
uuid: "3e573720b6184b95afa9a0d2734bd401"
format_stamp: "Formatted at 2026-05-02 07:57:18 on dist-test-slave-sj7c"
I20260502 07:57:18.137720 1126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/wal/instance:
uuid: "3e573720b6184b95afa9a0d2734bd401"
format_stamp: "Formatted at 2026-05-02 07:57:18 on dist-test-slave-sj7c"
I20260502 07:57:18.142927 1126 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260502 07:57:18.146131 1145 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:18.147516 1126 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260502 07:57:18.147704 1126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/wal
uuid: "3e573720b6184b95afa9a0d2734bd401"
format_stamp: "Formatted at 2026-05-02 07:57:18 on dist-test-slave-sj7c"
I20260502 07:57:18.147889 1126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:18.178797 1126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:18.179927 1126 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:18.180280 1126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:18.181421 1126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260502 07:57:18.183473 1126 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260502 07:57:18.183615 1126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:18.183779 1126 ts_tablet_manager.cc:616] Registered 0 tablets
I20260502 07:57:18.183894 1126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:18.237551 1126 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.2:44807
I20260502 07:57:18.237561 1257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.2:44807 every 8 connection(s)
I20260502 07:57:18.239292 1126 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/data/info.pb
I20260502 07:57:18.241837 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 1126
I20260502 07:57:18.242009 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-1/wal/instance
I20260502 07:57:18.250708 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.105.3:0
--local_ip_for_outbound_sockets=127.0.105.3
--webserver_interface=127.0.105.3
--webserver_port=0
--tserver_master_addrs=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260502 07:57:18.269603 1258 heartbeater.cc:344] Connected to a master server at 127.0.105.62:43729
I20260502 07:57:18.269943 1258 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:18.270889 1258 heartbeater.cc:507] Master 127.0.105.62:43729 requested a full tablet report, sending...
I20260502 07:57:18.272646 735 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:18.281060 1259 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:18.281222 1259 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:18.281723 1259 heartbeater.cc:507] Master 127.0.105.60:35545 requested a full tablet report, sending...
I20260502 07:57:18.282927 875 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:18.283205 1260 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:18.283367 1260 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:18.284376 1260 heartbeater.cc:507] Master 127.0.105.61:33363 requested a full tablet report, sending...
I20260502 07:57:18.285396 805 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:18.286320 805 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.105.2:32907
I20260502 07:57:18.300191 1120 heartbeater.cc:499] Master 127.0.105.61:33363 was elected leader, sending a full tablet report...
I20260502 07:57:18.543716 987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 51250627cebd4f9abefd40c111865403)
I20260502 07:57:18.543938 987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:18.544605 987 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
I20260502 07:57:18.545130 745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6" is_pre_election: true
I20260502 07:57:18.545409 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403" is_pre_election: true
I20260502 07:57:18.545513 745 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b49a409ffb84415ad9b53c0a862c470 in term 2.
I20260502 07:57:18.546051 859 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 364807e94fd04d41bfa5976f76f187a6; no voters: 51250627cebd4f9abefd40c111865403
I20260502 07:57:18.546307 987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260502 07:57:18.546438 987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 51250627cebd4f9abefd40c111865403)
I20260502 07:57:18.546558 987 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 2 FOLLOWER]: Advancing to term 3
I20260502 07:57:18.548816 987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:18.549418 987 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 3 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
I20260502 07:57:18.549901 745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6"
I20260502 07:57:18.549928 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403"
I20260502 07:57:18.550078 745 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 2 FOLLOWER]: Advancing to term 3
I20260502 07:57:18.552026 745 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b49a409ffb84415ad9b53c0a862c470 in term 3.
I20260502 07:57:18.552435 859 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 364807e94fd04d41bfa5976f76f187a6; no voters: 51250627cebd4f9abefd40c111865403
I20260502 07:57:18.552809 987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 3 FOLLOWER]: Leader election won for term 3
I20260502 07:57:18.553246 987 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 3 LEADER]: Becoming Leader. State: Replica: 0b49a409ffb84415ad9b53c0a862c470, State: Running, Role: LEADER
I20260502 07:57:18.553659 987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:18.555029 1265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b49a409ffb84415ad9b53c0a862c470. Latest consensus state: current_term: 3 leader_uuid: "0b49a409ffb84415ad9b53c0a862c470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:18.555436 1265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:18.556131 1266 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:18.558156 1266 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:18.558885 1266 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:18.558947 1266 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:18.560060 1266 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:18.561205 1266 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Loaded TSK: 0
I20260502 07:57:18.562044 1266 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:18.562729 1266 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:18.563411 1266 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
W20260502 07:57:18.598948 1263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:18.599273 1263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:18.599437 1263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:18.607553 1263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:18.607762 1263 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.105.3
I20260502 07:57:18.609546 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:18.609870 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:18.610327 745 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Rejecting Update request from peer 51250627cebd4f9abefd40c111865403 for earlier term 2. Current term is 3. Ops: []
I20260502 07:57:18.611224 1270 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260502 07:57:18.611590 1270 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 2 LEADER]: Stepping down as leader of term 2
I20260502 07:57:18.611688 1270 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:18.612016 1270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:18.612443 1270 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 2 FOLLOWER]: Advancing to term 3
I20260502 07:57:18.618566 1263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.105.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.105.3
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:18.620710 1263 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:18.622479 1263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:18.637888 1275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:18.639616 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:18.640516 1278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:18.641453 1263 server_base.cc:1061] running on GCE node
I20260502 07:57:18.642022 1263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:18.642850 1263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:18.644073 1263 hybrid_clock.cc:648] HybridClock initialized: now 1777708638644015 us; error 50 us; skew 500 ppm
I20260502 07:57:18.644384 1263 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:18.646525 1263 webserver.cc:492] Webserver started at http://127.0.105.3:34663/ using document root <none> and password file <none>
I20260502 07:57:18.647346 1263 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:18.647465 1263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:18.647835 1263 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260502 07:57:18.649996 1263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data/instance:
uuid: "d7984e7296e14b8592999e0897fc6a48"
format_stamp: "Formatted at 2026-05-02 07:57:18 on dist-test-slave-sj7c"
I20260502 07:57:18.650728 1263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/wal/instance:
uuid: "d7984e7296e14b8592999e0897fc6a48"
format_stamp: "Formatted at 2026-05-02 07:57:18 on dist-test-slave-sj7c"
I20260502 07:57:18.655738 1263 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260502 07:57:18.658814 1284 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:18.660116 1263 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260502 07:57:18.660275 1263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/wal
uuid: "d7984e7296e14b8592999e0897fc6a48"
format_stamp: "Formatted at 2026-05-02 07:57:18 on dist-test-slave-sj7c"
I20260502 07:57:18.660465 1263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:18.677099 1263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:18.677858 1263 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:18.678107 1263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:18.678966 1263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260502 07:57:18.680644 1263 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260502 07:57:18.680778 1263 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:18.680905 1263 ts_tablet_manager.cc:616] Registered 0 tablets
I20260502 07:57:18.680981 1263 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:18.715587 1263 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.3:44499
I20260502 07:57:18.715638 1396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.3:44499 every 8 connection(s)
I20260502 07:57:18.716950 1263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/data/info.pb
I20260502 07:57:18.726816 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 1263
I20260502 07:57:18.727082 420 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/ts-2/wal/instance
I20260502 07:57:18.735678 1399 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:18.735733 1398 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:18.736052 1398 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:18.736059 1399 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:18.737013 1398 heartbeater.cc:507] Master 127.0.105.60:35545 requested a full tablet report, sending...
I20260502 07:57:18.737042 1399 heartbeater.cc:507] Master 127.0.105.61:33363 requested a full tablet report, sending...
I20260502 07:57:18.738265 875 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:18.739240 804 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:18.740140 875 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.105.3:43543
I20260502 07:57:18.741617 1397 heartbeater.cc:344] Connected to a master server at 127.0.105.62:43729
I20260502 07:57:18.741743 1397 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:18.742026 1397 heartbeater.cc:507] Master 127.0.105.62:43729 requested a full tablet report, sending...
I20260502 07:57:18.742813 735 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:18.754856 420 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260502 07:57:18.789084 875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.788841 874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_f5b6e604b0284085b12dea20da69ca3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.788947 873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.788978 872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.790011 1260 heartbeater.cc:499] Master 127.0.105.61:33363 was elected leader, sending a full tablet report...
I20260502 07:57:18.796905 871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.796984 869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.796798 870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260502 07:57:18.798010 875 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_497a847e92ab43d5bd70df89906d8725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260502 07:57:18.811668 867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.811831 866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.811635 868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51778:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:18.825260 420 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu pid 841
I20260502 07:57:18.826512 905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260502 07:57:19.017155 815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 3 FOLLOWER]: Refusing update from remote peer 0b49a409ffb84415ad9b53c0a862c470: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260502 07:57:19.017598 420 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu with pid 841
I20260502 07:57:19.023592 745 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Refusing update from remote peer 0b49a409ffb84415ad9b53c0a862c470: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260502 07:57:19.050222 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.60:35545)
W20260502 07:57:19.053262 789 connection.cc:570] server connection from 127.0.0.1:35132 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 07:57:19.053664 720 connection.cc:570] server connection from 127.0.0.1:52506 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 07:57:19.054664 1410 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:19.051775 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.60:35545 (local address 127.0.0.1:51784)
0502 07:57:19.052113 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0502 07:57:19.052134 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0502 07:57:19.052346 (+ 212us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0502 07:57:19.053501 (+ 1155us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":94}
I20260502 07:57:19.081488 1445 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:19.081681 1445 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:19.082363 1445 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:19.082902 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403" is_pre_election: true
I20260502 07:57:19.083168 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 364807e94fd04d41bfa5976f76f187a6 in term 3.
I20260502 07:57:19.083748 722 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403; no voters:
I20260502 07:57:19.084064 1445 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260502 07:57:19.084206 1445 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:19.084304 1445 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 3 FOLLOWER]: Advancing to term 4
W20260502 07:57:19.084584 722 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:19.121151 1445 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:19.121937 1445 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 4 election: Requested vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:19.122692 815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403"
I20260502 07:57:19.122946 815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 3 FOLLOWER]: Advancing to term 4
I20260502 07:57:19.125846 815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 364807e94fd04d41bfa5976f76f187a6 in term 4.
I20260502 07:57:19.126818 722 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403; no voters:
I20260502 07:57:19.127285 1445 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 FOLLOWER]: Leader election won for term 4
W20260502 07:57:19.128858 722 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:19.141691 1445 raft_consensus.cc:697] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 LEADER]: Becoming Leader. State: Replica: 364807e94fd04d41bfa5976f76f187a6, State: Running, Role: LEADER
I20260502 07:57:19.142164 1445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:19.145941 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364807e94fd04d41bfa5976f76f187a6. Latest consensus state: current_term: 4 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:19.146238 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:19.146896 1450 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:19.149240 1450 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:19.150298 1450 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:19.150388 1450 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:19.151757 1450 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:19.152438 1450 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Loaded TSK: 0
I20260502 07:57:19.153040 1450 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:19.153735 1450 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:19.154393 1450 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:19.159474 734 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.159835 731 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_f5b6e604b0284085b12dea20da69ca3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.160060 728 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.159483 735 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.160346 730 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.160780 732 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.161007 733 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.161139 729 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.161526 726 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:19.161850 727 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52520:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260502 07:57:19.167696 731 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f5b6e604b0284085b12dea20da69ca3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260502 07:57:19.242547 1398 heartbeater.cc:499] Master 127.0.105.60:35545 was elected leader, sending a full tablet report...
W20260502 07:57:19.244051 1398 heartbeater.cc:646] Failed to heartbeat to 127.0.105.60:35545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:19.288750 1259 heartbeater.cc:499] Master 127.0.105.60:35545 was elected leader, sending a full tablet report...
W20260502 07:57:19.290365 1259 heartbeater.cc:646] Failed to heartbeat to 127.0.105.60:35545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:19.291493 1119 heartbeater.cc:499] Master 127.0.105.60:35545 was elected leader, sending a full tablet report...
W20260502 07:57:19.293119 1119 heartbeater.cc:646] Failed to heartbeat to 127.0.105.60:35545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:19.520218 815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 4 FOLLOWER]: Refusing update from remote peer 364807e94fd04d41bfa5976f76f187a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260502 07:57:19.521008 1448 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260502 07:57:19.530588 1455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364807e94fd04d41bfa5976f76f187a6. Latest consensus state: current_term: 4 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:19.530944 1455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:19.551286 1455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:19.551539 1455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
W20260502 07:57:19.555358 722 consensus_peers.cc:597] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 -> Peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Couldn't send request to peer 0b49a409ffb84415ad9b53c0a862c470. Status: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260502 07:57:19.555938 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "364807e94fd04d41bfa5976f76f187a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:19.556175 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:19.821105 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:35545
--webserver_interface=127.0.105.60
--webserver_port=44821
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 07:57:20.012601 731 catalog_manager.cc:2174] created HMS catalog entry for table [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
W20260502 07:57:20.021373 731 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.105.3:53835 (request call id 2) took 777 ms (client timeout 1000 ms). Trace:
W20260502 07:57:20.021580 731 rpcz_store.cc:269] 0502 07:57:19.243856 (+ 0us) service_pool.cc:167] Inserting onto call queue
0502 07:57:20.020552 (+776696us) service_pool.cc:224] Handling call
0502 07:57:20.021328 (+ 776us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":0}
I20260502 07:57:20.033677 731 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52520:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:20.068939 1193 tablet_service.cc:1511] Processing CreateTablet for tablet 2a935c8f7aeb41c1b5ed275cc3dd041b (DEFAULT_TABLE table=default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260502 07:57:20.071940 1193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a935c8f7aeb41c1b5ed275cc3dd041b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 07:57:20.084151 1053 tablet_service.cc:1511] Processing CreateTablet for tablet 2a935c8f7aeb41c1b5ed275cc3dd041b (DEFAULT_TABLE table=default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260502 07:57:20.086102 1053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a935c8f7aeb41c1b5ed275cc3dd041b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 07:57:20.088691 1332 tablet_service.cc:1511] Processing CreateTablet for tablet 2a935c8f7aeb41c1b5ed275cc3dd041b (DEFAULT_TABLE table=default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260502 07:57:20.090103 1332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a935c8f7aeb41c1b5ed275cc3dd041b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260502 07:57:20.108078 1473 tablet_bootstrap.cc:492] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401: Bootstrap starting.
I20260502 07:57:20.109138 1475 tablet_bootstrap.cc:492] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3: Bootstrap starting.
I20260502 07:57:20.112522 1475 tablet_bootstrap.cc:654] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3: Neither blocks nor log segments found. Creating new log.
I20260502 07:57:20.113032 1473 tablet_bootstrap.cc:654] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401: Neither blocks nor log segments found. Creating new log.
I20260502 07:57:20.113447 1474 tablet_bootstrap.cc:492] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48: Bootstrap starting.
I20260502 07:57:20.114346 1473 log.cc:826] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:20.117372 1474 tablet_bootstrap.cc:654] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48: Neither blocks nor log segments found. Creating new log.
I20260502 07:57:20.118712 1475 log.cc:826] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:20.119005 1474 log.cc:826] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:20.123584 1473 tablet_bootstrap.cc:492] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401: No bootstrap required, opened a new log
I20260502 07:57:20.123878 1473 ts_tablet_manager.cc:1403] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260502 07:57:20.124591 1475 tablet_bootstrap.cc:492] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3: No bootstrap required, opened a new log
I20260502 07:57:20.124835 1475 ts_tablet_manager.cc:1403] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20260502 07:57:20.125789 1474 tablet_bootstrap.cc:492] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48: No bootstrap required, opened a new log
I20260502 07:57:20.126133 1474 ts_tablet_manager.cc:1403] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260502 07:57:20.131525 1473 raft_consensus.cc:359] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
I20260502 07:57:20.131934 1473 raft_consensus.cc:385] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 07:57:20.131742 1475 raft_consensus.cc:359] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
I20260502 07:57:20.132048 1473 raft_consensus.cc:740] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e573720b6184b95afa9a0d2734bd401, State: Initialized, Role: FOLLOWER
I20260502 07:57:20.132071 1475 raft_consensus.cc:385] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 07:57:20.132149 1475 raft_consensus.cc:740] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3216d6df438a4041a7d61fb516f73ed3, State: Initialized, Role: FOLLOWER
I20260502 07:57:20.132794 1473 consensus_queue.cc:260] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
I20260502 07:57:20.132776 1475 consensus_queue.cc:260] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
I20260502 07:57:20.137804 1474 raft_consensus.cc:359] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
I20260502 07:57:20.138324 1474 raft_consensus.cc:385] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260502 07:57:20.138473 1474 raft_consensus.cc:740] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7984e7296e14b8592999e0897fc6a48, State: Initialized, Role: FOLLOWER
I20260502 07:57:20.139262 1474 consensus_queue.cc:260] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
I20260502 07:57:20.141373 1475 ts_tablet_manager.cc:1434] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
I20260502 07:57:20.141757 1118 heartbeater.cc:499] Master 127.0.105.62:43729 was elected leader, sending a full tablet report...
I20260502 07:57:20.151804 1258 heartbeater.cc:499] Master 127.0.105.62:43729 was elected leader, sending a full tablet report...
I20260502 07:57:20.152511 1473 ts_tablet_manager.cc:1434] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401: Time spent starting tablet: real 0.028s user 0.018s sys 0.008s
I20260502 07:57:20.154495 1397 heartbeater.cc:499] Master 127.0.105.62:43729 was elected leader, sending a full tablet report...
I20260502 07:57:20.155264 1474 ts_tablet_manager.cc:1434] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48: Time spent starting tablet: real 0.029s user 0.008s sys 0.015s
W20260502 07:57:20.200649 734 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c61458705db241fd8d267928ef573ae4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.200981 728 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_97e488bcfc6344bb9ab4a680e6cbc086 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.201978 735 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0b3858b7ca5141aebae79d45a08c7f7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.202250 733 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_66ca50c5f9f94d8aa108db59823fb91a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.203195 729 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fd6af681f6ad416ea58e484e70699dfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.203472 726 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d32274c9b0c84d3dadb672c216102926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.203197 727 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0fe7ca1b362f4a8e9c78d28d1531545b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.200649 730 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_497a847e92ab43d5bd70df89906d8725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260502 07:57:20.201973 732 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_517dadf96c184083819e9a71e8de9d2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260502 07:57:20.208595 1481 raft_consensus.cc:493] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:20.208858 1481 raft_consensus.cc:515] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
F20260502 07:57:20.210882 944 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 2 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f809e94fdcd google::LogMessage::Fail() at ??:0
@ 0x7f809e953b93 google::LogMessage::SendToLog() at ??:0
I20260502 07:57:20.215777 1481 leader_election.cc:290] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807), 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
@ 0x7f809e94f7cc google::LogMessage::Flush() at ??:0
W20260502 07:57:20.221731 1400 tablet.cc:2404] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7f809e94fa11 google::LogMessage::~LogMessage() at ??:0
I20260502 07:57:20.224208 1213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a935c8f7aeb41c1b5ed275cc3dd041b" candidate_uuid: "d7984e7296e14b8592999e0897fc6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e573720b6184b95afa9a0d2734bd401" is_pre_election: true
I20260502 07:57:20.224741 1213 raft_consensus.cc:2468] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7984e7296e14b8592999e0897fc6a48 in term 0.
I20260502 07:57:20.225538 1285 leader_election.cc:304] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e573720b6184b95afa9a0d2734bd401, d7984e7296e14b8592999e0897fc6a48; no voters:
I20260502 07:57:20.226461 1481 raft_consensus.cc:2804] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260502 07:57:20.226751 1481 raft_consensus.cc:493] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:20.226866 1481 raft_consensus.cc:3060] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 0 FOLLOWER]: Advancing to term 1
W20260502 07:57:20.240744 1121 tablet.cc:2404] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7f80b5428574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f80b5426e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260502 07:57:20.251142 1073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a935c8f7aeb41c1b5ed275cc3dd041b" candidate_uuid: "d7984e7296e14b8592999e0897fc6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3216d6df438a4041a7d61fb516f73ed3" is_pre_election: true
I20260502 07:57:20.251688 1073 raft_consensus.cc:2468] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7984e7296e14b8592999e0897fc6a48 in term 0.
@ 0x7f80b542ea9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f80b542e919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7f80a16e8529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f80a4cd66db start_thread at ??:0
@ 0x7f809d80061f clone at ??:0
W20260502 07:57:20.273520 1264 tablet.cc:2404] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260502 07:57:20.285494 1463 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:20.285900 1463 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:20.286015 1463 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260502 07:57:20.298661 1481 raft_consensus.cc:515] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
I20260502 07:57:20.299540 1481 leader_election.cc:290] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [CANDIDATE]: Term 1 election: Requested vote from peers 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807), 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
W20260502 07:57:20.300807 1463 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:20.300900 1463 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:20.300988 1463 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:20.301056 1463 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:20.301110 1213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a935c8f7aeb41c1b5ed275cc3dd041b" candidate_uuid: "d7984e7296e14b8592999e0897fc6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e573720b6184b95afa9a0d2734bd401"
I20260502 07:57:20.301311 1213 raft_consensus.cc:3060] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401 [term 0 FOLLOWER]: Advancing to term 1
I20260502 07:57:20.304452 1073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a935c8f7aeb41c1b5ed275cc3dd041b" candidate_uuid: "d7984e7296e14b8592999e0897fc6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3216d6df438a4041a7d61fb516f73ed3"
I20260502 07:57:20.304657 1073 raft_consensus.cc:3060] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260502 07:57:20.319005 1463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:35545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=44821
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:20.321074 1463 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:20.323477 1463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:20.338207 1496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:20.338496 1497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:20.342134 1213 raft_consensus.cc:2468] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3e573720b6184b95afa9a0d2734bd401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7984e7296e14b8592999e0897fc6a48 in term 1.
I20260502 07:57:20.342887 1285 leader_election.cc:304] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e573720b6184b95afa9a0d2734bd401, d7984e7296e14b8592999e0897fc6a48; no voters:
I20260502 07:57:20.343349 1481 raft_consensus.cc:2804] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260502 07:57:20.343876 1073 raft_consensus.cc:2468] T 2a935c8f7aeb41c1b5ed275cc3dd041b P 3216d6df438a4041a7d61fb516f73ed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7984e7296e14b8592999e0897fc6a48 in term 1.
I20260502 07:57:20.346822 1481 raft_consensus.cc:697] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [term 1 LEADER]: Becoming Leader. State: Replica: d7984e7296e14b8592999e0897fc6a48, State: Running, Role: LEADER
I20260502 07:57:20.347402 1481 consensus_queue.cc:237] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 } } peers { permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 } } peers { permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" member_type: VOTER last_known_addr { host: "127.0.105.3" port: 44499 } }
W20260502 07:57:20.347661 1499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:20.349687 1463 server_base.cc:1061] running on GCE node
I20260502 07:57:20.350814 1463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:20.355803 1463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:20.357221 1463 hybrid_clock.cc:648] HybridClock initialized: now 1777708640357152 us; error 64 us; skew 500 ppm
I20260502 07:57:20.357661 1463 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:20.365409 1463 webserver.cc:492] Webserver started at http://127.0.105.60:44821/ using document root <none> and password file <none>
I20260502 07:57:20.366555 1463 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:20.366707 1463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:20.375514 1463 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260502 07:57:20.379112 1510 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:20.380947 1463 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260502 07:57:20.381119 1463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
uuid: "0b49a409ffb84415ad9b53c0a862c470"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:20.381865 1463 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:20.399665 1463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:20.400578 1463 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:20.400877 1463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:20.422147 1463 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:35545
I20260502 07:57:20.422176 1561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:35545 every 8 connection(s)
I20260502 07:57:20.423839 1463 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
I20260502 07:57:20.428047 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 1463
I20260502 07:57:20.435882 1562 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:20.440681 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Bootstrap starting.
I20260502 07:57:20.450912 1562 log.cc:826] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:20.461380 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 07:57:20.462064 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Bootstrap complete.
I20260502 07:57:20.469727 1562 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:20.470822 1562 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b49a409ffb84415ad9b53c0a862c470, State: Initialized, Role: FOLLOWER
I20260502 07:57:20.471534 1562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:20.472378 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:20.472720 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:20.473826 1562 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:20.489107 1581 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:20.489199 1581 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: loading cluster ID for follower catalog manager: success
I20260502 07:57:20.492899 1581 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: acquiring CA information for follower catalog manager: success
I20260502 07:57:20.495332 1581 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:20.538049 1584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 364807e94fd04d41bfa5976f76f187a6)
I20260502 07:57:20.538259 1584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:20.539103 1584 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:20.548064 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:20.548696 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 3.
I20260502 07:57:20.549268 792 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters:
I20260502 07:57:20.549574 1584 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260502 07:57:20.549705 1584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 364807e94fd04d41bfa5976f76f187a6)
I20260502 07:57:20.549806 1584 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 4 FOLLOWER]: Advancing to term 5
W20260502 07:57:20.554885 1404 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.62:43729)
W20260502 07:57:20.555233 1404 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.62:43729)
W20260502 07:57:20.555853 1404 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.62:43729)
W20260502 07:57:20.558362 789 connection.cc:570] client connection to 127.0.105.62:43729 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 07:57:20.559688 1461 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.555273 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.62:43729 (local address 127.0.105.1:37749)
0502 07:57:20.555841 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0502 07:57:20.555861 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0502 07:57:20.555980 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0502 07:57:20.558611 (+ 2631us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":460}
W20260502 07:57:20.560600 1583 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.488387 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.62:43729 (local address 127.0.0.1:52530)
0502 07:57:20.489057 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0502 07:57:20.489095 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0502 07:57:20.489247 (+ 152us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0502 07:57:20.559245 (+ 69998us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":328,"threads_started":1}
W20260502 07:57:20.560978 1459 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.556682 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.62:43729 (local address 127.0.105.3:48607)
0502 07:57:20.556929 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0502 07:57:20.556946 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0502 07:57:20.557056 (+ 110us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0502 07:57:20.560257 (+ 3201us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132}
W20260502 07:57:20.562266 1587 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.557017 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.62:43729 (local address 127.0.0.1:52532)
0502 07:57:20.561844 (+ 4827us) negotiation.cc:107] Waiting for socket to connect
0502 07:57:20.561967 (+ 123us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":4664,"threads_started":1}
W20260502 07:57:20.562404 1460 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.557385 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.62:43729 (local address 127.0.105.2:49807)
0502 07:57:20.557620 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0502 07:57:20.557637 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0502 07:57:20.557753 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0502 07:57:20.561631 (+ 3878us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127}
I20260502 07:57:20.560987 1463 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20260502 07:57:20.567163 1463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 07:57:20.568586 789 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:20.580018 1584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:20.580747 1584 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 5 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:20.581881 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470"
I20260502 07:57:20.582084 1537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 3 FOLLOWER]: Advancing to term 5
W20260502 07:57:20.582491 789 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:20.630009 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 5.
I20260502 07:57:20.630756 792 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:20.631274 1584 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Leader election won for term 5
I20260502 07:57:20.632457 1584 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:20.632822 1584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:20.635846 1589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 5 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:20.636247 1589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:20.639616 1590 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:20.641909 1590 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:20.643400 1590 tablet_loader.cc:96] loaded metadata for tablet 2a935c8f7aeb41c1b5ed275cc3dd041b (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:20.643651 1590 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:20.644454 1590 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:20.644538 1590 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:20.645690 1590 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:20.646765 1590 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:20.647341 1590 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:20.648098 1590 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:20.648905 1590 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:20.658314 1119 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:20.658511 1119 heartbeater.cc:499] Master 127.0.105.60:35545 was elected leader, sending a full tablet report...
I20260502 07:57:20.659204 805 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:35144
I20260502 07:57:20.660532 803 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:35144
I20260502 07:57:20.660912 802 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:35144
I20260502 07:57:20.661180 801 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:35144
I20260502 07:57:20.661423 800 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:35144
I20260502 07:57:20.661695 1527 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:33543; Asking this server to re-register.
I20260502 07:57:20.662503 1119 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:20.662644 797 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35144:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:20.664193 1527 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:20.662940 796 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35144:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:20.662343 798 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35144:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:20.662032 799 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35144:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:20.658345 804 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35144:
name: "default.Table_b0a63cd3a6fb4f1d86da504a21e8516f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260502 07:57:20.669745 789 consensus_peers.cc:597] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 -> Peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Couldn't send request to peer 364807e94fd04d41bfa5976f76f187a6. Status: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20260502 07:57:20.670529 984 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
I20260502 07:57:20.670644 1537 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Refusing update from remote peer 51250627cebd4f9abefd40c111865403: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 10. (index mismatch)
*** Check failure stack trace: ***
I20260502 07:57:20.671559 1589 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.000s
I20260502 07:57:20.671763 1589 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Peer 0b49a409ffb84415ad9b53c0a862c470 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
@ 0x7f7469336dcd google::LogMessage::Fail() at ??:0
@ 0x7f746933ab93 google::LogMessage::SendToLog() at ??:0
I20260502 07:57:20.675508 1259 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:20.675678 1259 heartbeater.cc:499] Master 127.0.105.60:35545 was elected leader, sending a full tablet report...
I20260502 07:57:20.676054 1537 pending_rounds.cc:85] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Aborting all ops after (but not including) 5
I20260502 07:57:20.676271 1537 pending_rounds.cc:107] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260502 07:57:20.676427 1537 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7f74693367cc google::LogMessage::Flush() at ??:0
@ 0x7f7469336a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f747fe0f574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f747fe0de21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f747fe15a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f747fe15919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
I20260502 07:57:20.708130 1527 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:33797; Asking this server to re-register.
I20260502 07:57:20.709056 1596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:20.709383 1596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:20.709889 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 5 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:20.710122 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:20.710513 1259 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:20.712317 1527 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
@ 0x7f746c0cf529 kudu::Thread::SuperviseThread() at ??:0
I20260502 07:57:20.714903 1593 mvcc.cc:204] Tried to move back new op lower bound from 7281494589550317568 to 7281494585929723904. Current Snapshot: MvccSnapshot[applied={T|T < 7281494589550317568 or (T in {7281494589550317568})}]
I20260502 07:57:20.715747 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:20.715911 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:20.725188 1584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:20.725427 1584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:20.726214 805 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Sending DeleteTablet for 3 replicas of tablet 2a935c8f7aeb41c1b5ed275cc3dd041b
I20260502 07:57:20.727574 802 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35144:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:20.728382 801 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35144:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:20.729090 800 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35144:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f746f6bd6db start_thread at ??:0
@ 0x7f74681e761f clone at ??:0
I20260502 07:57:20.856287 1398 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:20.856534 1398 heartbeater.cc:499] Master 127.0.105.60:35545 was elected leader, sending a full tablet report...
I20260502 07:57:20.858412 1527 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:38271; Asking this server to re-register.
I20260502 07:57:20.859189 1398 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:20.860641 1527 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
W20260502 07:57:20.902704 1288 connection.cc:570] client connection to 127.0.105.61:33363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 07:57:20.903002 1469 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.731588 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.0.105.1:41957)
0502 07:57:20.731747 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0502 07:57:20.731753 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0502 07:57:20.902737 (+170984us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52242: BlockingRecv error: recv got EOF from 127.0.0.1:52242 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260502 07:57:20.904047 1608 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.731243 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.105.2:44807)
0502 07:57:20.733836 (+ 2593us) server_negotiation.cc:207] Beginning negotiation
0502 07:57:20.733842 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0502 07:57:20.733859 (+ 17us) server_negotiation.cc:408] Connection header received
0502 07:57:20.733938 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0502 07:57:20.733944 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0502 07:57:20.734012 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0502 07:57:20.734102 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0502 07:57:20.903710 (+169608us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55696: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":2496,"thread_start_us":3037,"threads_started":1}
W20260502 07:57:20.905267 1587 negotiation.cc:336] Failed RPC negotiation. Trace:
0502 07:57:20.904576 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.0.105.61:33363 (local address 127.0.0.1:35160)
0502 07:57:20.904976 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0502 07:57:20.905065 (+ 89us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":242}
I20260502 07:57:21.153095 1120 heartbeater.cc:499] Master 127.0.105.61:33363 was elected leader, sending a full tablet report...
I20260502 07:57:21.215693 1566 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 51250627cebd4f9abefd40c111865403)
I20260502 07:57:21.215878 1566 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:21.217563 1566 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:21.218502 1511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:21.219481 1514 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:21.219696 1511 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:21.219884 1511 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:21.220196 1566 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260502 07:57:21.223304 1614 consensus_queue.cc:1048] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" member_type: VOTER last_known_addr { host: "127.0.105.2" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260502 07:57:21.233397 1618 consensus_queue.cc:1048] T 2a935c8f7aeb41c1b5ed275cc3dd041b P d7984e7296e14b8592999e0897fc6a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" member_type: VOTER last_known_addr { host: "127.0.105.1" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260502 07:57:21.823220 1628 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 51250627cebd4f9abefd40c111865403)
I20260502 07:57:21.823410 1628 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:21.824127 1628 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:21.825999 1511 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:21.826154 1514 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:21.826292 1514 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:21.826540 1628 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260502 07:57:22.417634 1637 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 51250627cebd4f9abefd40c111865403)
I20260502 07:57:22.417841 1637 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:22.418483 1637 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:22.420590 1514 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:22.420778 1511 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:22.420912 1511 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:22.421118 1637 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260502 07:57:23.159912 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--webserver_interface=127.0.105.61
--webserver_port=40445
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 07:57:23.325593 1651 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 51250627cebd4f9abefd40c111865403)
I20260502 07:57:23.325785 1651 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:23.326433 1651 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:23.328478 1511 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:23.328823 1514 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:23.328946 1514 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:23.329187 1651 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260502 07:57:23.477139 1648 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:23.477456 1648 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:23.477587 1648 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:23.485407 1648 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:23.485507 1648 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:23.485621 1648 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:23.485705 1648 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:23.494984 1648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=40445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:23.496599 1648 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:23.498281 1648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:23.505959 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:23.505992 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:23.506754 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:23.507136 1648 server_base.cc:1061] running on GCE node
I20260502 07:57:23.507869 1648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:23.509104 1648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:23.510325 1648 hybrid_clock.cc:648] HybridClock initialized: now 1777708643510295 us; error 45 us; skew 500 ppm
I20260502 07:57:23.510632 1648 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:23.512557 1648 webserver.cc:492] Webserver started at http://127.0.105.61:40445/ using document root <none> and password file <none>
I20260502 07:57:23.513231 1648 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:23.513381 1648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:23.518237 1648 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260502 07:57:23.520802 1668 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:23.522264 1648 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260502 07:57:23.522423 1648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:23.523103 1648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:23.541728 1648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:23.542578 1648 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:23.542876 1648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:23.561560 1648 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:33363
I20260502 07:57:23.561595 1719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:33363 every 8 connection(s)
I20260502 07:57:23.562978 1648 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
I20260502 07:57:23.566900 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 1648
I20260502 07:57:23.570176 1720 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:23.574757 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap starting.
I20260502 07:57:23.589051 1720 log.cc:826] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:23.594297 1399 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:23.604717 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260502 07:57:23.605230 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap complete.
I20260502 07:57:23.609918 1720 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:23.610462 1720 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Initialized, Role: FOLLOWER
I20260502 07:57:23.611013 1720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:23.611781 1727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:23.612027 1727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:23.612638 1720 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:23.626879 1741 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:23.627075 1741 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: loading cluster ID for follower catalog manager: success
I20260502 07:57:23.623656 1648 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:23.629300 1648 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:23.631174 1741 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: acquiring CA information for follower catalog manager: success
I20260502 07:57:23.633474 1741 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:23.753297 1120 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:23.753500 1120 heartbeater.cc:499] Master 127.0.105.61:33363 was elected leader, sending a full tablet report...
I20260502 07:57:23.755671 1685 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:39495; Asking this server to re-register.
I20260502 07:57:23.756464 1120 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:23.758278 1685 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:23.762224 1260 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:23.763475 1685 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:50871; Asking this server to re-register.
I20260502 07:57:23.764074 1260 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:23.765391 1685 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:23.995019 1727 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:23.995311 1727 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:23.997036 1727 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:23.997742 1669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:23.999503 1669 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:24.003566 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:24.003811 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 5.
I20260502 07:57:24.004540 1672 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:24.004902 1727 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260502 07:57:24.005016 1727 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:24.005144 1727 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 5 FOLLOWER]: Advancing to term 6
I20260502 07:57:24.008653 1727 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:24.009205 1727 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 6 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:24.009697 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470"
I20260502 07:57:24.009935 1537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 5 FOLLOWER]: Advancing to term 6
W20260502 07:57:24.010416 1669 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:24.011790 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 6.
I20260502 07:57:24.012120 1672 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:24.012356 1727 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Leader election won for term 6
I20260502 07:57:24.012748 1727 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:24.013170 1727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:24.015854 1751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 6 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:24.016113 1751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:24.016768 1753 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:24.019136 1753 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:24.020746 1753 tablet_loader.cc:96] loaded metadata for tablet b110d76eba324930b224b28d6c88897d (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:24.021040 1753 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:24.021788 1753 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:24.021889 1753 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:24.022954 1753 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:24.023810 1753 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:24.024569 1753 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:24.025293 1753 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:24.026073 1753 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:24.060694 1684 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.060969 1682 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.061247 1683 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.062143 1678 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.062218 1679 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.062322 1677 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.061901 1685 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.062711 1680 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:24.063257 1676 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35874:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:24.063196 1681 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35874:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 07:57:24.067755 1740 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fcf2fc9bdcd google::LogMessage::Fail() at ??:0
@ 0x7fcf2fc9fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fcf2fc9b7cc google::LogMessage::Flush() at ??:0
@ 0x7fcf2fc9ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fcf46774574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fcf46772e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fcf4677aa9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fcf4677a919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fcf32a34529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcf360226db start_thread at ??:0
@ 0x7fcf2eb4c61f clone at ??:0
W20260502 07:57:24.260180 1407 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.61:33363)
W20260502 07:57:24.260764 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:24.261030 1407 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:33363)
I20260502 07:57:25.568463 1463 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:25.568666 1463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:26.242002 1771 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:26.242208 1771 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:26.242960 1771 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:26.243773 1511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20260502 07:57:26.244931 1514 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:26.245193 1511 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:26.245280 1511 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:26.245575 1771 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260502 07:57:26.500855 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--webserver_interface=127.0.105.61
--webserver_port=40445
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 07:57:26.829413 1775 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:26.829733 1775 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:26.829846 1775 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:26.837559 1775 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:26.837633 1775 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:26.837743 1775 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:26.837932 1775 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:26.847635 1775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=40445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:26.849120 1775 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:26.850742 1775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:26.858783 1782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:26.858881 1784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:26.858951 1781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:26.860149 1775 server_base.cc:1061] running on GCE node
I20260502 07:57:26.860831 1775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:26.862135 1775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:26.863337 1775 hybrid_clock.cc:648] HybridClock initialized: now 1777708646863301 us; error 31 us; skew 500 ppm
I20260502 07:57:26.863663 1775 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:26.865712 1775 webserver.cc:492] Webserver started at http://127.0.105.61:40445/ using document root <none> and password file <none>
I20260502 07:57:26.866343 1775 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:26.866426 1775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:26.872156 1775 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260502 07:57:26.874984 1790 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:26.876253 1775 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260502 07:57:26.876425 1775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:26.877208 1775 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:26.907755 1775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:26.908654 1775 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:26.908978 1775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:26.933130 1775 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:33363
I20260502 07:57:26.933148 1841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:33363 every 8 connection(s)
I20260502 07:57:26.934741 1775 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
I20260502 07:57:26.941201 1842 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:26.943394 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 1775
I20260502 07:57:26.944762 1842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap starting.
I20260502 07:57:26.953292 1842 log.cc:826] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:26.964350 1842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260502 07:57:26.964916 1842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap complete.
I20260502 07:57:26.969794 1842 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:26.970577 1842 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Initialized, Role: FOLLOWER
I20260502 07:57:26.971266 1842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 6.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:26.971997 1846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:26.972286 1846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:26.972859 1842 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:26.984198 1860 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:26.984340 1860 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: loading cluster ID for follower catalog manager: success
I20260502 07:57:26.988343 1860 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: acquiring CA information for follower catalog manager: success
I20260502 07:57:26.990690 1860 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:26.991779 1775 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:26.997174 1775 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:27.274487 1399 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:27.278755 1807 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:47157; Asking this server to re-register.
I20260502 07:57:27.279922 1399 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:27.280340 1120 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:27.281948 1806 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:27.282251 1807 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:33599; Asking this server to re-register.
I20260502 07:57:27.283215 1120 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:27.283663 1260 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:27.284452 1806 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:27.284691 1807 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:37109; Asking this server to re-register.
I20260502 07:57:27.285181 1260 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:27.286172 1807 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:27.353601 1846 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:27.353845 1846 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:27.355949 1846 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:27.356978 1791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:27.358989 1791 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:27.362473 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:27.362648 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 6.
I20260502 07:57:27.363246 1794 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:27.363551 1846 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260502 07:57:27.363684 1846 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:27.363788 1846 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 6 FOLLOWER]: Advancing to term 7
I20260502 07:57:27.446777 1846 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:27.447686 1846 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 7 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:27.448449 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470"
I20260502 07:57:27.448660 1537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 6 FOLLOWER]: Advancing to term 7
W20260502 07:57:27.449246 1791 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:27.550204 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 7.
I20260502 07:57:27.550798 1794 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:27.551353 1846 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Leader election won for term 7
I20260502 07:57:27.551811 1846 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:27.552424 1846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 6.11, Last appended by leader: 11, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:27.554884 1870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 7 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:27.555119 1870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:27.555821 1872 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:27.558416 1872 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:27.560262 1872 tablet_loader.cc:96] loaded metadata for tablet b110d76eba324930b224b28d6c88897d (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:27.560523 1872 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:27.561336 1872 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:27.561405 1872 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:27.562474 1872 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:27.563326 1872 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:27.564169 1872 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:27.564983 1872 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:27.565762 1872 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:27.633630 1807 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44272
I20260502 07:57:27.633795 1806 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44272
I20260502 07:57:27.633944 1805 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44272
I20260502 07:57:27.634143 1804 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44272
I20260502 07:57:27.634779 1802 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44272
I20260502 07:57:27.634730 1803 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_b0a63cd3a6fb4f1d86da504a21e8516f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.635388 1802 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.635525 1806 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.634881 1800 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.635974 1804 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.636876 1807 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.637717 1801 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44272:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:27.637790 1799 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.638365 1798 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:27.638463 1805 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 07:57:27.642153 1859 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fdce1f26dcd google::LogMessage::Fail() at ??:0
@ 0x7fdce1f2ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7fdce1f267cc google::LogMessage::Flush() at ??:0
@ 0x7fdce1f26a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fdcf89ff574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fdcf89fde21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fdcf8a05a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fdcf8a05919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fdce4cbf529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdce82ad6db start_thread at ??:0
@ 0x7fdce0dd761f clone at ??:0
W20260502 07:57:27.829599 1149 connection.cc:570] client connection to 127.0.105.61:33363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 07:57:27.829625 1009 connection.cc:570] client connection to 127.0.105.61:33363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 07:57:27.829771 1288 connection.cc:570] client connection to 127.0.105.61:33363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260502 07:57:27.830543 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:27.831337 1407 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:27.839280 1407 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.61:33363)
I20260502 07:57:29.992205 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--webserver_interface=127.0.105.61
--webserver_port=40445
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 07:57:30.322413 1891 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:30.322700 1891 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:30.322880 1891 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:30.331496 1891 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:30.331596 1891 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:30.331713 1891 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:30.331791 1891 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:30.341925 1891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=40445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:30.343557 1891 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:30.345319 1891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:30.352994 1896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:30.353221 1897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:30.353951 1899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:30.354101 1891 server_base.cc:1061] running on GCE node
I20260502 07:57:30.354851 1891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:30.356225 1891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:30.357400 1891 hybrid_clock.cc:648] HybridClock initialized: now 1777708650357345 us; error 47 us; skew 500 ppm
I20260502 07:57:30.357786 1891 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:30.359913 1891 webserver.cc:492] Webserver started at http://127.0.105.61:40445/ using document root <none> and password file <none>
I20260502 07:57:30.360580 1891 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:30.360668 1891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:30.365695 1891 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260502 07:57:30.368366 1905 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:30.369673 1891 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260502 07:57:30.369868 1891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:30.370539 1891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:30.393181 1891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:30.393970 1891 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:30.394263 1891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:30.413036 1891 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:33363
I20260502 07:57:30.413092 1956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:33363 every 8 connection(s)
I20260502 07:57:30.414577 1891 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
I20260502 07:57:30.418670 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 1891
I20260502 07:57:30.421464 1957 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:30.425251 1957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap starting.
I20260502 07:57:30.434513 1957 log.cc:826] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:30.445206 1957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260502 07:57:30.445678 1957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap complete.
I20260502 07:57:30.450493 1957 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:30.451335 1957 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Initialized, Role: FOLLOWER
I20260502 07:57:30.452190 1957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 7.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:30.453341 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:30.453676 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:30.454434 1957 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:30.467345 1975 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:30.467458 1975 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: loading cluster ID for follower catalog manager: success
I20260502 07:57:30.472435 1975 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: acquiring CA information for follower catalog manager: success
I20260502 07:57:30.475602 1975 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:30.476770 1891 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:30.481843 1891 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:30.570175 1463 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:30.570369 1463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:30.843958 1120 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:30.843993 1260 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:30.845568 1399 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:30.846544 1922 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:33181; Asking this server to re-register.
I20260502 07:57:30.846587 1921 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:47387; Asking this server to re-register.
I20260502 07:57:30.846694 1920 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:36351; Asking this server to re-register.
I20260502 07:57:30.847241 1399 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:30.847426 1260 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:30.847927 1120 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:30.848987 1922 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:30.849112 1921 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:30.849244 1920 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:30.926225 1961 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:30.926496 1961 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:30.928408 1961 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:30.928890 1906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:30.930452 1906 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:30.935034 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 8 candidate_status { last_received { term: 7 index: 12 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:30.935357 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 7.
I20260502 07:57:30.936015 1909 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:30.936334 1961 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260502 07:57:30.936445 1961 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:30.936578 1961 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 7 FOLLOWER]: Advancing to term 8
I20260502 07:57:30.940069 1961 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:30.940659 1961 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 8 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:30.941102 1537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 8 candidate_status { last_received { term: 7 index: 12 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470"
I20260502 07:57:30.941308 1537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 7 FOLLOWER]: Advancing to term 8
W20260502 07:57:30.941893 1906 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:30.943154 1537 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 8.
I20260502 07:57:30.943634 1909 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:30.943959 1961 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Leader election won for term 8
I20260502 07:57:30.944306 1961 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:30.944741 1961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.12, Last appended by leader: 12, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:30.947091 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 8 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:30.947309 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:30.947960 1988 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:30.950320 1988 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:30.952052 1988 tablet_loader.cc:96] loaded metadata for tablet b110d76eba324930b224b28d6c88897d (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:30.952390 1988 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:30.953202 1988 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:30.953261 1988 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:30.954164 1988 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:30.954843 1988 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:30.955681 1988 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:30.956367 1988 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:30.957041 1988 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:30.992895 1920 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.993398 1916 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.993502 1915 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.992838 1922 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.993011 1919 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.994059 1917 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.994560 1918 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.994792 1913 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44968:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:30.995359 1921 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:30.995157 1914 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44968:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 07:57:30.998620 1974 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f01ead07dcd google::LogMessage::Fail() at ??:0
@ 0x7f01ead0bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f01ead077cc google::LogMessage::Flush() at ??:0
@ 0x7f01ead07a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f02017e0574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f02017dee21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f02017e6a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f02017e6919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7f01edaa0529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f01f108e6db start_thread at ??:0
@ 0x7f01e9bb861f clone at ??:0
W20260502 07:57:31.178017 1407 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.61:33363)
W20260502 07:57:31.178653 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:31.178889 1407 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:33363)
I20260502 07:57:31.643036 2002 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:31.643476 2002 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:31.644109 2002 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:31.645020 1511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:57:31.646087 1514 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:31.646332 1511 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:31.646432 1511 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:31.646646 2002 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260502 07:57:32.624891 420 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu pid 1463
I20260502 07:57:32.631464 1557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260502 07:57:32.740167 420 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu with pid 1463
I20260502 07:57:33.324052 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:35545
--webserver_interface=127.0.105.60
--webserver_port=44821
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 07:57:33.643290 2012 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:33.643559 2012 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:33.643666 2012 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:33.652137 2012 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:33.652223 2012 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:33.652283 2012 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:33.652330 2012 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:33.662101 2012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.60:35545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.105.60
--webserver_port=44821
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:33.663589 2012 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:33.665419 2012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:33.672837 2017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:33.673014 2018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:33.673220 2020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:33.673827 2012 server_base.cc:1061] running on GCE node
I20260502 07:57:33.674479 2012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:33.675733 2012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:33.676970 2012 hybrid_clock.cc:648] HybridClock initialized: now 1777708653676938 us; error 45 us; skew 500 ppm
I20260502 07:57:33.677309 2012 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:33.679728 2012 webserver.cc:492] Webserver started at http://127.0.105.60:44821/ using document root <none> and password file <none>
I20260502 07:57:33.680574 2012 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:33.680733 2012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:33.686632 2012 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260502 07:57:33.689359 2026 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:33.690837 2012 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260502 07:57:33.690969 2012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
uuid: "0b49a409ffb84415ad9b53c0a862c470"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:33.691681 2012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:33.702837 2012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:33.703637 2012 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:33.703860 2012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:33.722296 2012 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.60:35545
I20260502 07:57:33.722369 2077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.60:35545 every 8 connection(s)
I20260502 07:57:33.723889 2012 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-2/data/info.pb
I20260502 07:57:33.730168 2078 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:33.731284 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 2012
I20260502 07:57:33.733827 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Bootstrap starting.
I20260502 07:57:33.743993 2078 log.cc:826] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:33.755730 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=1 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260502 07:57:33.756419 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Bootstrap complete.
I20260502 07:57:33.761840 2078 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:33.762624 2078 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b49a409ffb84415ad9b53c0a862c470, State: Initialized, Role: FOLLOWER
I20260502 07:57:33.763278 2078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:33.763995 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:33.764210 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:33.764950 2078 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:33.771449 2097 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260502 07:57:33.774704 2012 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:33.779597 2012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:33.787753 1119 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:33.789860 2043 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:40237; Asking this server to re-register.
I20260502 07:57:33.790405 1119 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:33.791775 2043 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:33.802556 1259 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:33.803490 2043 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:44627; Asking this server to re-register.
I20260502 07:57:33.803920 1259 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:33.804895 2043 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:34.122990 1398 heartbeater.cc:344] Connected to a master server at 127.0.105.60:35545
I20260502 07:57:34.124109 2043 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:45065; Asking this server to re-register.
I20260502 07:57:34.124593 1398 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:34.125603 2043 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:34.280303 2100 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:34.280576 2100 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:34.282701 2100 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:34.283427 2027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:34.284761 2030 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:34.284976 2027 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:34.285140 2027 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:34.285471 2100 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260502 07:57:34.776728 2097 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:34.776875 2097 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: loading cluster ID for follower catalog manager: success
I20260502 07:57:34.780843 2097 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: acquiring CA information for follower catalog manager: success
I20260502 07:57:34.783268 2097 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:34.985461 2106 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:34.985685 2106 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:34.986595 2106 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:34.989065 2027 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:34.989241 2030 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:34.989377 2030 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:34.989732 2106 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260502 07:57:35.522017 2110 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:35.522192 2110 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:35.522775 2110 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:35.524819 2030 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:35.525022 2027 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:35.525151 2027 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:35.525398 2110 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260502 07:57:36.074249 2115 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:36.074450 2115 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:36.075034 2115 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:36.077201 2030 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:36.077382 2027 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:36.077517 2027 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470; no voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403
I20260502 07:57:36.077775 2115 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260502 07:57:36.734694 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--webserver_interface=127.0.105.61
--webserver_port=40445
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 07:57:37.048719 2120 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:37.049043 2120 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:37.049203 2120 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:37.057651 2120 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:37.057732 2120 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:37.057819 2120 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:37.057914 2120 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:37.068122 2120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=40445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:37.069705 2120 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:37.071460 2120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:37.079258 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:37.079474 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:37.080272 2128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:37.080662 2120 server_base.cc:1061] running on GCE node
I20260502 07:57:37.081290 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:37.082516 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:37.083757 2120 hybrid_clock.cc:648] HybridClock initialized: now 1777708657083710 us; error 35 us; skew 500 ppm
I20260502 07:57:37.084127 2120 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:37.086117 2120 webserver.cc:492] Webserver started at http://127.0.105.61:40445/ using document root <none> and password file <none>
I20260502 07:57:37.086822 2120 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:37.086937 2120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:37.091934 2120 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260502 07:57:37.094468 2134 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:37.095703 2120 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260502 07:57:37.095873 2120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:37.096503 2120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:37.120370 2120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:37.121119 2120 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:37.121387 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:37.141072 2120 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:33363
I20260502 07:57:37.141112 2185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:33363 every 8 connection(s)
I20260502 07:57:37.142532 2120 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
I20260502 07:57:37.143639 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 2120
I20260502 07:57:37.149598 2186 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:37.152904 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap starting.
I20260502 07:57:37.161734 2186 log.cc:826] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:37.173703 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260502 07:57:37.174351 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap complete.
I20260502 07:57:37.179342 2186 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:37.180130 2186 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Initialized, Role: FOLLOWER
I20260502 07:57:37.180588 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 8.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:37.181222 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:37.181456 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:37.182119 2186 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:37.193265 2204 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:37.193372 2204 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: loading cluster ID for follower catalog manager: success
I20260502 07:57:37.197089 2204 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: acquiring CA information for follower catalog manager: success
I20260502 07:57:37.199591 2204 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:37.194303 2120 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:37.200104 2120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:37.375707 1399 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:37.378757 2151 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:42479; Asking this server to re-register.
I20260502 07:57:37.380266 1399 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:37.382658 2151 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:37.383244 1260 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:37.384699 2151 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:43443; Asking this server to re-register.
I20260502 07:57:37.385218 1260 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:37.385799 1120 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:37.386303 2151 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:37.386946 2150 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:52207; Asking this server to re-register.
I20260502 07:57:37.387398 1120 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:37.388494 2150 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:37.578640 2190 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:37.578926 2190 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:37.580993 2190 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:37.581527 2135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
W20260502 07:57:37.583323 2135 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:37.587703 2053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 9 candidate_status { last_received { term: 8 index: 13 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:37.588055 2053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 8.
I20260502 07:57:37.588663 2138 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:37.589018 2190 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260502 07:57:37.589183 2190 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:37.589287 2190 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 8 FOLLOWER]: Advancing to term 9
I20260502 07:57:37.592693 2190 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:37.593235 2190 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 9 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:37.593768 2053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 9 candidate_status { last_received { term: 8 index: 13 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470"
I20260502 07:57:37.593981 2053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 8 FOLLOWER]: Advancing to term 9
W20260502 07:57:37.594483 2135 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111)
I20260502 07:57:37.597476 2053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 9.
I20260502 07:57:37.597816 2138 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403; no voters: 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:37.598083 2190 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 FOLLOWER]: Leader election won for term 9
I20260502 07:57:37.598441 2190 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:37.598843 2190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.13, Last appended by leader: 13, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:37.601298 2214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 9 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:37.601547 2214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:37.602145 2216 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:37.604439 2216 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:37.606024 2216 tablet_loader.cc:96] loaded metadata for tablet b110d76eba324930b224b28d6c88897d (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:37.606261 2216 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:37.607013 2216 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:37.607136 2216 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:37.608178 2216 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:37.608961 2216 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:37.609724 2216 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:37.610472 2216 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:37.611423 2216 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:37.615372 2150 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:42552
I20260502 07:57:37.615456 2151 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:42552
I20260502 07:57:37.615854 2149 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:42552
I20260502 07:57:37.616008 2147 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:42552
I20260502 07:57:37.616326 2148 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:42552
I20260502 07:57:37.616848 2151 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.616813 2144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.617713 2145 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42552:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:37.617285 2147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.619813 2150 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.617812 2142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.619760 2143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.620559 2146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_b0a63cd3a6fb4f1d86da504a21e8516f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.621148 2149 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:37.621623 2148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42552:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 07:57:37.622782 2203 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fbeb83a4dcd google::LogMessage::Fail() at ??:0
@ 0x7fbeb83a8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbeb83a47cc google::LogMessage::Flush() at ??:0
@ 0x7fbeb83a4a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fbecee7d574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fbecee7be21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fbecee83a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fbecee83919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fbebb13d529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbebe72b6db start_thread at ??:0
@ 0x7fbeb725561f clone at ??:0
W20260502 07:57:37.807788 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:37.808701 1407 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:37.821031 1407 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.61:33363)
I20260502 07:57:38.781055 2012 master_runner.cc:186] Error getting master registration for 127.0.105.62:43729: Network error: Client connection negotiation failed: client connection to 127.0.105.62:43729: connect: Connection refused (error 111),
I20260502 07:57:38.781286 2012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:40.077819 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:43729
--webserver_interface=127.0.105.62
--webserver_port=42221
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 07:57:40.389284 2232 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:40.389648 2232 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:40.389804 2232 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:40.398694 2232 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:40.398779 2232 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:40.398840 2232 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:40.398880 2232 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:40.408505 2232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.62:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.105.62
--webserver_port=42221
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:40.410004 2232 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:40.411756 2232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:40.420053 2238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:40.420089 2239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:40.421334 2241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:40.422428 2232 server_base.cc:1061] running on GCE node
I20260502 07:57:40.423277 2232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:40.424610 2232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:40.425793 2232 hybrid_clock.cc:648] HybridClock initialized: now 1777708660425758 us; error 27 us; skew 500 ppm
I20260502 07:57:40.426151 2232 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:40.428184 2232 webserver.cc:492] Webserver started at http://127.0.105.62:42221/ using document root <none> and password file <none>
I20260502 07:57:40.428916 2232 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:40.429050 2232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:40.434232 2232 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260502 07:57:40.437049 2247 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:40.438364 2232 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260502 07:57:40.438526 2232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
uuid: "364807e94fd04d41bfa5976f76f187a6"
format_stamp: "Formatted at 2026-05-02 07:57:15 on dist-test-slave-sj7c"
I20260502 07:57:40.439245 2232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:40.459937 2232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:40.460764 2232 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:40.461072 2232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:40.481069 2232 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.62:43729
I20260502 07:57:40.481088 2298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.62:43729 every 8 connection(s)
I20260502 07:57:40.482647 2232 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-0/data/info.pb
I20260502 07:57:40.484483 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 2232
I20260502 07:57:40.490700 2299 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:40.494709 2299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Bootstrap starting.
I20260502 07:57:40.504050 2299 log.cc:826] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:40.513276 2299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=1 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260502 07:57:40.513773 2299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: Bootstrap complete.
I20260502 07:57:40.518576 2299 raft_consensus.cc:359] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:40.519296 2299 raft_consensus.cc:740] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364807e94fd04d41bfa5976f76f187a6, State: Initialized, Role: FOLLOWER
I20260502 07:57:40.519922 2299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:40.520640 2304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:40.520869 2304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:40.521546 2299 sys_catalog.cc:565] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:40.543274 2318 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:40.543387 2318 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: loading cluster ID for follower catalog manager: success
I20260502 07:57:40.546555 2318 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: acquiring CA information for follower catalog manager: success
I20260502 07:57:40.540634 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:57:40.546785 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:40.548560 2318 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:40.631775 1397 heartbeater.cc:344] Connected to a master server at 127.0.105.62:43729
I20260502 07:57:40.631994 1397 heartbeater.cc:499] Master 127.0.105.62:43729 was elected leader, sending a full tablet report...
I20260502 07:57:40.635011 2263 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:38271; Asking this server to re-register.
I20260502 07:57:40.635792 1397 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:40.637423 2263 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:40.805977 1118 heartbeater.cc:344] Connected to a master server at 127.0.105.62:43729
I20260502 07:57:40.806221 1118 heartbeater.cc:499] Master 127.0.105.62:43729 was elected leader, sending a full tablet report...
I20260502 07:57:40.806483 1258 heartbeater.cc:344] Connected to a master server at 127.0.105.62:43729
I20260502 07:57:40.806648 1258 heartbeater.cc:499] Master 127.0.105.62:43729 was elected leader, sending a full tablet report...
I20260502 07:57:40.807596 2263 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:47441; Asking this server to re-register.
I20260502 07:57:40.807922 2264 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:50905; Asking this server to re-register.
I20260502 07:57:40.808171 1118 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:40.808496 1258 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:40.809494 2264 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:40.809782 2263 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:40.952798 2304 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:40.953047 2304 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:40.954759 2304 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:40.955490 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:40.957204 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:40.960754 2053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "0b49a409ffb84415ad9b53c0a862c470" is_pre_election: true
I20260502 07:57:40.961125 2053 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 364807e94fd04d41bfa5976f76f187a6 for earlier term 5. Current term is 9.
I20260502 07:57:40.961959 2251 leader_election.cc:400] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 364807e94fd04d41bfa5976f76f187a6 for earlier term 5. Current term is 9.
I20260502 07:57:40.962070 2251 leader_election.cc:403] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260502 07:57:40.962354 2304 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 4 FOLLOWER]: Advancing to term 9
I20260502 07:57:40.966176 2304 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 9 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 364807e94fd04d41bfa5976f76f187a6 for earlier term 5. Current term is 9.
I20260502 07:57:41.242319 2325 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:41.242501 2325 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:41.243331 2325 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
W20260502 07:57:41.245030 2030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20260502 07:57:41.246688 2030 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:41.250769 2274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 10 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6" is_pre_election: true
I20260502 07:57:41.251211 2274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b49a409ffb84415ad9b53c0a862c470 in term 9.
I20260502 07:57:41.251789 2027 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 364807e94fd04d41bfa5976f76f187a6; no voters: 51250627cebd4f9abefd40c111865403
I20260502 07:57:41.252130 2325 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260502 07:57:41.252254 2325 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:41.252362 2325 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 9 FOLLOWER]: Advancing to term 10
I20260502 07:57:41.254292 2325 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:41.254843 2325 leader_election.cc:290] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 10 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363)
I20260502 07:57:41.255282 2274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b49a409ffb84415ad9b53c0a862c470" candidate_term: 10 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6"
I20260502 07:57:41.255479 2274 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 9 FOLLOWER]: Advancing to term 10
W20260502 07:57:41.256072 2030 leader_election.cc:336] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:57:41.257469 2274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b49a409ffb84415ad9b53c0a862c470 in term 10.
I20260502 07:57:41.257839 2027 leader_election.cc:304] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b49a409ffb84415ad9b53c0a862c470, 364807e94fd04d41bfa5976f76f187a6; no voters: 51250627cebd4f9abefd40c111865403
I20260502 07:57:41.258086 2325 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 10 FOLLOWER]: Leader election won for term 10
I20260502 07:57:41.258476 2325 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [term 10 LEADER]: Becoming Leader. State: Replica: 0b49a409ffb84415ad9b53c0a862c470, State: Running, Role: LEADER
I20260502 07:57:41.258927 2325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:41.261615 2329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b49a409ffb84415ad9b53c0a862c470. Latest consensus state: current_term: 10 leader_uuid: "0b49a409ffb84415ad9b53c0a862c470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:41.261895 2329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:41.262517 2331 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:41.264734 2331 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:41.266508 2331 tablet_loader.cc:96] loaded metadata for tablet b110d76eba324930b224b28d6c88897d (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:41.266753 2331 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:41.267568 2331 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:41.267637 2331 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:41.268728 2331 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:41.269476 2331 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 0b49a409ffb84415ad9b53c0a862c470: Loaded TSK: 0
I20260502 07:57:41.270264 2331 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:41.270949 2331 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:41.271673 2331 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:41.295194 2042 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.295249 2041 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.296610 2043 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.296687 2037 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.296787 2040 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.297202 2038 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.297372 2039 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.297240 2036 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:41.297765 2034 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53878:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:41.298064 2035 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53878:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 07:57:41.301221 2096 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fa697464dcd google::LogMessage::Fail() at ??:0
@ 0x7fa697468b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa6974647cc google::LogMessage::Flush() at ??:0
@ 0x7fa697464a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fa6adf3d574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fa6adf3be21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fa6adf43a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fa6adf43919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fa69a1fd529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa69d7eb6db start_thread at ??:0
@ 0x7fa69631561f clone at ??:0
W20260502 07:57:41.484380 1407 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.60:35545)
W20260502 07:57:41.485013 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.60:35545)
W20260502 07:57:41.485240 1407 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.60:35545)
I20260502 07:57:41.878559 2348 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:41.878763 2348 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:41.879675 2348 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:41.881844 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:41.882061 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:41.882161 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:41.882433 2348 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260502 07:57:43.005100 2355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:43.005260 2355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:43.005754 2355 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:43.008047 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:43.008246 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:43.008322 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:43.008505 2355 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260502 07:57:43.559616 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--webserver_interface=127.0.105.61
--webserver_port=40445
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260502 07:57:43.872525 2359 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:43.872836 2359 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:43.872973 2359 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:43.881884 2359 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:43.881958 2359 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:43.882051 2359 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:43.882139 2359 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:43.892158 2359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=40445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:43.893728 2359 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:43.895473 2359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:43.903681 2366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:43.905133 2365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:43.905331 2368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:43.906349 2359 server_base.cc:1061] running on GCE node
I20260502 07:57:43.907121 2359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:43.908447 2359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:43.909672 2359 hybrid_clock.cc:648] HybridClock initialized: now 1777708663909638 us; error 50 us; skew 500 ppm
I20260502 07:57:43.910022 2359 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:43.912130 2359 webserver.cc:492] Webserver started at http://127.0.105.61:40445/ using document root <none> and password file <none>
I20260502 07:57:43.912869 2359 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:43.913005 2359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:43.917655 2374 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:43.917819 2374 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:43.918092 2359 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260502 07:57:43.918429 2374 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:43.920346 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:43.920533 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:43.920658 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:43.920903 2374 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260502 07:57:43.921519 2377 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:43.922801 2359 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260502 07:57:43.923209 2359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:43.923856 2359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:43.935945 2359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:43.936707 2359 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:43.937009 2359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:43.956683 2359 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:33363
I20260502 07:57:43.956713 2428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:33363 every 8 connection(s)
I20260502 07:57:43.958158 2359 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
I20260502 07:57:43.965610 2429 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:43.966284 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 2359
I20260502 07:57:43.969884 2429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap starting.
I20260502 07:57:43.978677 2429 log.cc:826] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:43.990566 2429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20260502 07:57:43.991252 2429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap complete.
I20260502 07:57:43.996222 2429 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:43.997149 2429 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Initialized, Role: FOLLOWER
I20260502 07:57:43.997785 2429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 9.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:43.998510 2433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:43.998731 2433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:43.999557 2429 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:44.010816 2447 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:44.011034 2447 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: loading cluster ID for follower catalog manager: success
I20260502 07:57:44.015930 2447 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: acquiring CA information for follower catalog manager: success
I20260502 07:57:44.020929 2447 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:44.043530 2359 master_runner.cc:186] Error getting master registration for 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111),
I20260502 07:57:44.047190 2359 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:44.405215 1399 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:44.407734 2394 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:44755; Asking this server to re-register.
I20260502 07:57:44.408470 1399 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:44.410116 1260 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:44.410151 2394 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:44.410708 1120 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:44.411402 2394 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:51601; Asking this server to re-register.
I20260502 07:57:44.411877 1260 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:44.411968 2393 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:43437; Asking this server to re-register.
I20260502 07:57:44.412472 1120 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:44.413058 2393 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:44.413585 2394 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:44.566676 2454 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:44.566946 2454 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:44.568789 2454 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:44.569272 2274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 10 candidate_status { last_received { term: 9 index: 14 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6" is_pre_election: true
I20260502 07:57:44.569571 2274 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51250627cebd4f9abefd40c111865403 in current term 10: Already voted for candidate 0b49a409ffb84415ad9b53c0a862c470 in this term.
W20260502 07:57:44.569765 2381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
W20260502 07:57:44.571115 2381 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:44.571259 2381 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51250627cebd4f9abefd40c111865403; no voters: 0b49a409ffb84415ad9b53c0a862c470, 364807e94fd04d41bfa5976f76f187a6
I20260502 07:57:44.571564 2454 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 9 FOLLOWER]: Advancing to term 10
I20260502 07:57:44.574781 2454 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260502 07:57:45.079006 2460 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:45.079263 2460 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:45.079983 2460 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:45.080516 2274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 11 candidate_status { last_received { term: 9 index: 14 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6" is_pre_election: true
I20260502 07:57:45.080752 2274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 10.
I20260502 07:57:45.081275 2378 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403; no voters:
I20260502 07:57:45.081591 2460 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260502 07:57:45.081734 2460 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:45.081851 2460 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 10 FOLLOWER]: Advancing to term 11
W20260502 07:57:45.082090 2381 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:45.083859 2460 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:45.084388 2460 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 11 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:45.084808 2274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 11 candidate_status { last_received { term: 9 index: 14 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6"
I20260502 07:57:45.085027 2274 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 10 FOLLOWER]: Advancing to term 11
W20260502 07:57:45.085572 2381 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:45.086970 2274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 11.
I20260502 07:57:45.087383 2378 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403; no voters: 0b49a409ffb84415ad9b53c0a862c470
I20260502 07:57:45.087678 2460 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Leader election won for term 11
I20260502 07:57:45.088064 2460 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:45.088534 2460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.14, Last appended by leader: 14, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:45.091032 2463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 11 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:45.091291 2463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:45.091871 2465 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:45.094017 2465 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:45.095713 2465 tablet_loader.cc:96] loaded metadata for tablet b110d76eba324930b224b28d6c88897d (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:45.095978 2465 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:45.096678 2465 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:45.096771 2465 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:45.097812 2465 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:45.098526 2465 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:45.099421 2465 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:45.100190 2465 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:45.101022 2465 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:45.125409 2393 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44030
I20260502 07:57:45.125540 2394 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44030
I20260502 07:57:45.125913 2391 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44030
I20260502 07:57:45.125980 2392 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44030
I20260502 07:57:45.126235 2390 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a935c8f7aeb41c1b5ed275cc3dd041b from {username='slave'} at 127.0.0.1:44030
I20260502 07:57:45.127230 2393 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.128620 2394 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.127571 2392 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.128564 2387 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.128988 2388 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44030:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:45.127688 2389 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_b0a63cd3a6fb4f1d86da504a21e8516f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.127002 2390 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.129446 2385 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.127785 2386 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:45.127954 2391 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44030:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 07:57:45.133165 2446 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fd1157e6dcd google::LogMessage::Fail() at ??:0
@ 0x7fd1157eab93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd1157e67cc google::LogMessage::Flush() at ??:0
@ 0x7fd1157e6a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd12c2bf574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fd12c2bde21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fd12c2c5a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fd12c2c5919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fd11857f529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd11bb6d6db start_thread at ??:0
@ 0x7fd11469761f clone at ??:0
W20260502 07:57:45.329633 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:45.330585 1407 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:45.333032 1407 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.61:33363)
I20260502 07:57:45.549149 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:57:45.549340 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:46.734210 420 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
/tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--webserver_interface=127.0.105.61
--webserver_port=40445
--builtin_ntp_servers=127.0.105.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260502 07:57:47.008543 2483 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:47.008749 2483 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
W20260502 07:57:47.010838 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260502 07:57:47.011214 2483 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:47.012480 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:47.012651 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:47.012771 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:47.013057 2483 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260502 07:57:47.056227 2482 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260502 07:57:47.056564 2482 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260502 07:57:47.056691 2482 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260502 07:57:47.065559 2482 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260502 07:57:47.065670 2482 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260502 07:57:47.065786 2482 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260502 07:57:47.065863 2482 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260502 07:57:47.076385 2482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.105.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37631
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.105.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.105.61
--webserver_port=40445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 02 May 2026 07:43:22 UTC on bdcb31816ec0
build id 11706
ASAN enabled
I20260502 07:57:47.078052 2482 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260502 07:57:47.079856 2482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260502 07:57:47.087601 2490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:47.087572 2491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260502 07:57:47.088276 2493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260502 07:57:47.088798 2482 server_base.cc:1061] running on GCE node
I20260502 07:57:47.089643 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260502 07:57:47.091189 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260502 07:57:47.092412 2482 hybrid_clock.cc:648] HybridClock initialized: now 1777708667092383 us; error 46 us; skew 500 ppm
I20260502 07:57:47.092741 2482 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260502 07:57:47.094869 2482 webserver.cc:492] Webserver started at http://127.0.105.61:40445/ using document root <none> and password file <none>
I20260502 07:57:47.095736 2482 fs_manager.cc:362] Metadata directory not provided
I20260502 07:57:47.095875 2482 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260502 07:57:47.100824 2482 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260502 07:57:47.103403 2499 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260502 07:57:47.104650 2482 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260502 07:57:47.104813 2482 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data,/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
uuid: "51250627cebd4f9abefd40c111865403"
format_stamp: "Formatted at 2026-05-02 07:57:16 on dist-test-slave-sj7c"
I20260502 07:57:47.105430 2482 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260502 07:57:47.117343 2482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260502 07:57:47.118070 2482 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260502 07:57:47.118346 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260502 07:57:47.137632 2482 rpc_server.cc:307] RPC server started. Bound to: 127.0.105.61:33363
I20260502 07:57:47.137660 2550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.105.61:33363 every 8 connection(s)
I20260502 07:57:47.139204 2482 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/master-1/data/info.pb
I20260502 07:57:47.141656 420 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXQ2lRx/build/asan/bin/kudu as pid 2482
I20260502 07:57:47.146212 2551 sys_catalog.cc:263] Verifying existing consensus state
I20260502 07:57:47.149420 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap starting.
I20260502 07:57:47.157883 2551 log.cc:826] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Log is configured to *not* fsync() on all Append() calls
I20260502 07:57:47.168607 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260502 07:57:47.169162 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Bootstrap complete.
I20260502 07:57:47.173867 2551 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:47.174670 2551 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51250627cebd4f9abefd40c111865403, State: Initialized, Role: FOLLOWER
I20260502 07:57:47.175287 2551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 11.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:47.175964 2555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 11 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:47.176201 2555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: FOLLOWER
I20260502 07:57:47.176873 2551 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: configured and running, proceeding with master startup.
I20260502 07:57:47.201887 2569 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:47.201997 2569 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: loading cluster ID for follower catalog manager: success
I20260502 07:57:47.205080 2569 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: acquiring CA information for follower catalog manager: success
I20260502 07:57:47.207342 2569 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260502 07:57:47.210004 2482 master_runner.cc:186] Error getting master registration for 127.0.105.60:35545: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111),
I20260502 07:57:47.213539 2482 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:47.426239 1399 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:47.428930 2516 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d7984e7296e14b8592999e0897fc6a48" instance_seqno: 1777708638707791) as {username='slave'} at 127.0.105.3:46495; Asking this server to re-register.
I20260502 07:57:47.429785 1399 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:47.431530 1120 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:47.431660 1260 heartbeater.cc:344] Connected to a master server at 127.0.105.61:33363
I20260502 07:57:47.431833 2516 ts_manager.cc:194] Registered new tserver with Master: d7984e7296e14b8592999e0897fc6a48 (127.0.105.3:44499)
I20260502 07:57:47.432857 2516 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3216d6df438a4041a7d61fb516f73ed3" instance_seqno: 1777708637726123) as {username='slave'} at 127.0.105.1:56657; Asking this server to re-register.
I20260502 07:57:47.432875 2515 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3e573720b6184b95afa9a0d2734bd401" instance_seqno: 1777708638225207) as {username='slave'} at 127.0.105.2:50083; Asking this server to re-register.
I20260502 07:57:47.433393 1120 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:47.433470 1260 heartbeater.cc:461] Registering TS with master...
I20260502 07:57:47.434590 2516 ts_manager.cc:194] Registered new tserver with Master: 3216d6df438a4041a7d61fb516f73ed3 (127.0.105.1:41957)
I20260502 07:57:47.434720 2515 ts_manager.cc:194] Registered new tserver with Master: 3e573720b6184b95afa9a0d2734bd401 (127.0.105.2:44807)
I20260502 07:57:47.535540 2576 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:47.535706 2576 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:47.536324 2576 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:47.540814 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:47.543692 2526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "364807e94fd04d41bfa5976f76f187a6" candidate_term: 12 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "51250627cebd4f9abefd40c111865403" is_pre_election: true
I20260502 07:57:47.544505 2526 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 364807e94fd04d41bfa5976f76f187a6 for term 12 because replica has last-logged OpId of term: 11 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 8.
I20260502 07:57:47.545127 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:47.545405 2576 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260502 07:57:47.660280 2555 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:47.660593 2555 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:47.662351 2555 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:47.662890 2274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 12 candidate_status { last_received { term: 11 index: 15 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6" is_pre_election: true
W20260502 07:57:47.662957 2503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:47.663142 2274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 11.
I20260502 07:57:47.663667 2500 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403; no voters:
I20260502 07:57:47.664065 2555 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260502 07:57:47.664220 2555 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260502 07:57:47.664331 2555 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 11 FOLLOWER]: Advancing to term 12
W20260502 07:57:47.664558 2503 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:47.667927 2555 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:47.668475 2555 leader_election.cc:290] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 12 election: Requested vote from peers 364807e94fd04d41bfa5976f76f187a6 (127.0.105.62:43729), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
I20260502 07:57:47.668833 2274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51250627cebd4f9abefd40c111865403" candidate_term: 12 candidate_status { last_received { term: 11 index: 15 } } ignore_live_leader: false dest_uuid: "364807e94fd04d41bfa5976f76f187a6"
I20260502 07:57:47.669040 2274 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 11 FOLLOWER]: Advancing to term 12
W20260502 07:57:47.669698 2503 leader_election.cc:336] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:47.670914 2274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51250627cebd4f9abefd40c111865403 in term 12.
I20260502 07:57:47.671275 2500 leader_election.cc:304] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6, 51250627cebd4f9abefd40c111865403; no voters: 0b49a409ffb84415ad9b53c0a862c470
I20260502 07:57:47.671592 2555 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 12 FOLLOWER]: Leader election won for term 12
I20260502 07:57:47.671885 2555 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [term 12 LEADER]: Becoming Leader. State: Replica: 51250627cebd4f9abefd40c111865403, State: Running, Role: LEADER
I20260502 07:57:47.672284 2555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.15, Last appended by leader: 15, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:47.674567 2581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51250627cebd4f9abefd40c111865403. Latest consensus state: current_term: 12 leader_uuid: "51250627cebd4f9abefd40c111865403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } } }
I20260502 07:57:47.674825 2581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403 [sys.catalog]: This master's current role is: LEADER
I20260502 07:57:47.675480 2583 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260502 07:57:47.677765 2583 catalog_manager.cc:679] Loaded metadata for table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a]
I20260502 07:57:47.679546 2583 tablet_loader.cc:96] loaded metadata for tablet b110d76eba324930b224b28d6c88897d (table default.table_f5b6e604b0284085b12dea20da69ca3d [id=4b9e5ad8064b4321adff4ab9b0b6f71a])
I20260502 07:57:47.679800 2583 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260502 07:57:47.680604 2583 catalog_manager.cc:1269] Loaded cluster ID: 6e5c6bc9d7d5475e9b313d55c0071796
I20260502 07:57:47.680697 2583 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260502 07:57:47.681761 2583 catalog_manager.cc:1514] Loading token signing keys...
I20260502 07:57:47.682526 2583 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 51250627cebd4f9abefd40c111865403: Loaded TSK: 0
I20260502 07:57:47.683458 2583 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260502 07:57:47.684242 2583 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260502 07:57:47.684919 2583 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260502 07:57:47.725749 2515 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_fd6af681f6ad416ea58e484e70699dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.726507 2516 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_c61458705db241fd8d267928ef573ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.727167 2511 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_497a847e92ab43d5bd70df89906d8725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.730829 2507 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47448:
table { table_name: "default.Table_f5b6e604b0284085b12dea20da69ca3d" } new_table_name: "default.Table_6b72e1b16b9e4929bc8149e6fbb8ae65" modify_external_catalogs: true
I20260502 07:57:47.729941 2510 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_97e488bcfc6344bb9ab4a680e6cbc086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.729977 2509 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_0fe7ca1b362f4a8e9c78d28d1531545b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.729378 2514 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_0b3858b7ca5141aebae79d45a08c7f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.732527 2513 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_66ca50c5f9f94d8aa108db59823fb91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.727648 2512 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_517dadf96c184083819e9a71e8de9d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260502 07:57:47.732694 2508 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47448:
name: "default.Table_d32274c9b0c84d3dadb672c216102926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260502 07:57:47.735795 2568 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_497a847e92ab43d5bd70df89906d8725, message={"server":"","servicePrincipal":"","db":"default","table":"table_497a847e92ab43d5bd70df89906d8725","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_497a847e92ab43d5bd70df89906d8725\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708638},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_497a847e92ab43d5bd70df89906d8725\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_497a847e92ab43d5bd70df89906d8725\",\"kudu.table_id\":\"bef5b64eb069479792dbbe1e9d75f12b\",\"transient_lastDdlTime\":\"1777708638\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777708639, eventType=CREATE_TABLE, dbName=default, tableName=table_f5b6e604b0284085b12dea20da69ca3d, message={"server":"","servicePrincipal":"","db":"default","table":"table_f5b6e604b0284085b12dea20da69ca3d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f5b6e604b0284085b12dea20da69ca3d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777708639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXQ2lRx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1777708620760110-420-0/minicluster-data/warehouse/table_f5b6e604b0284085b12dea20da69ca3d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f5b6e604b0284085b12dea20da69ca3d\",\"kudu.table_id\":\"4b9e5ad8064b4321adff4ab9b0b6f71a\",\"transient_lastDdlTime\":\"1777708639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"6e5c6bc9d7d5475e9b313d55c0071796\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777708639,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fc357161dcd google::LogMessage::Fail() at ??:0
@ 0x7fc357165b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc3571617cc google::LogMessage::Flush() at ??:0
@ 0x7fc357161a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fc36dc3a574 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fc36dc38e21 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fc36dc40a9c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fc36dc40919 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xaa4cad std::function<>::operator()() at ??:?
@ 0x7fc359efa529 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc35d4e86db start_thread at ??:0
@ 0x7fc35601261f clone at ??:0
W20260502 07:57:47.926314 1407 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.105.61:33363)
W20260502 07:57:47.926904 1407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.105.61:33363)
W20260502 07:57:47.927085 1407 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.105.61:33363)
I20260502 07:57:49.579449 420 master-stress-test.cc:577] Killing test thread 1/25
I20260502 07:57:49.978256 2598 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:49.978485 2598 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:49.979192 2598 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:49.982782 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:49.983018 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:49.983165 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:49.983435 2598 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:57:50.551829 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:57:50.552033 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:52.995278 2606 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:52.995478 2606 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:52.995988 2606 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:52.997179 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260502 07:57:53.000144 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:53.000391 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:53.000502 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:53.000769 2606 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:57:55.554482 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:57:55.554683 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:57:58.395838 2617 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:57:58.396032 2617 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:57:58.396632 2617 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:57:58.397732 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:57:58.399906 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:57:58.400159 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:57:58.400261 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:57:58.400552 2617 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:58:00.557286 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:00.557451 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:02.074910 2626 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:58:02.075160 2626 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:58:02.075850 2626 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:58:02.078173 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:02.078336 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:58:02.078421 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:58:02.078617 2626 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:58:05.559662 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:05.559834 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:10.562346 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:10.562521 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:15.564750 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:15.564967 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 07:58:19.505916 1399 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (63 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:19.506016 1260 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (63 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:19.505996 1120 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (63 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:58:20.567466 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:20.567624 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:21.739910 2655 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:58:21.740157 2655 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:58:21.740957 2655 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:58:21.742363 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 07:58:21.744352 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:21.744571 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:58:21.744709 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:58:21.744944 2655 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:58:25.569644 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:25.569828 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:28.671976 2673 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:58:28.672142 2673 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:58:28.672698 2673 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:58:28.673812 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:58:28.677285 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:28.677486 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:58:28.677603 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:58:28.677810 2673 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:58:30.572239 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:30.572431 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:33.662130 2684 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:58:33.662338 2684 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:58:33.662930 2684 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:58:33.665750 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:33.665946 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:58:33.666042 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:58:33.666297 2684 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:58:35.574683 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:35.574877 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:40.577152 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:40.577332 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:41.291160 2701 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:58:41.291304 2701 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:58:41.291841 2701 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:58:41.292845 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 07:58:41.296135 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:41.296352 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:58:41.296423 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:58:41.296681 2701 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:58:45.579492 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:45.579634 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:47.800427 2716 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:58:47.800590 2716 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:58:47.801144 2716 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:58:47.802196 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:58:47.804034 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:58:47.804245 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:58:47.804320 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:58:47.804497 2716 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:58:50.581928 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:50.582139 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:58:55.585911 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:58:55.586113 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:00.588876 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:00.589082 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:04.637902 2744 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:04.638100 2744 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:04.639093 2744 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:04.640087 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:59:04.641131 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:04.641284 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:04.641361 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:04.641611 2744 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:05.591573 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:05.591789 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:10.593758 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:10.593933 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:15.601384 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:15.601560 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260502 07:59:19.642186 1120 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (183 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:19.642256 1399 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (183 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:19.642433 1260 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (183 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 07:59:20.604322 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:20.604526 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:20.621889 2771 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:20.622093 2771 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:20.623102 2771 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:20.623729 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:59:20.625499 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:20.625715 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:20.625833 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:20.626545 2771 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:21.761848 2775 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:21.762019 2775 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:21.762589 2775 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:21.765401 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:21.765619 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:21.765762 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:21.766007 2775 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:25.167178 2784 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:25.167378 2784 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:25.168012 2784 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:25.172245 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:25.172477 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:25.172580 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:25.172834 2784 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:25.606945 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:25.607201 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:30.609615 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:30.609848 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:33.783536 2800 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:33.783782 2800 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:33.784601 2800 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:33.785768 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260502 07:59:33.787160 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:33.787367 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:33.787459 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:33.787814 2800 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:35.612605 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:35.612761 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:40.615190 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:40.615345 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:40.618281 2253 maintenance_manager.cc:419] P 364807e94fd04d41bfa5976f76f187a6: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033360
I20260502 07:59:40.658824 2252 maintenance_manager.cc:643] P 364807e94fd04d41bfa5976f76f187a6: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.016s sys 0.004s Metrics: {"bytes_written":6565,"cfile_init":1,"dirs.queue_time_us":458,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1034,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":5,"thread_start_us":499,"threads_started":2,"wal-append.queue_time_us":247}
I20260502 07:59:40.659888 2253 maintenance_manager.cc:419] P 364807e94fd04d41bfa5976f76f187a6: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 275 bytes on disk
I20260502 07:59:40.660490 2252 maintenance_manager.cc:643] P 364807e94fd04d41bfa5976f76f187a6: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20260502 07:59:42.466706 2818 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:42.466946 2818 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:42.467708 2818 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:42.468878 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:59:42.472283 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:42.472530 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:42.472637 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:42.472926 2818 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:45.617903 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:45.618059 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:49.363981 2831 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:49.364189 2831 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:49.364895 2831 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:49.366034 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:59:49.368006 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:49.368225 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:49.368321 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:49.368587 2831 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:50.620522 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:50.620699 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 07:59:54.597126 2844 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 07:59:54.597335 2844 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 07:59:54.597949 2844 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 07:59:54.599220 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 07:59:54.604175 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 07:59:54.604423 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 07:59:54.604521 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 07:59:54.604763 2844 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 07:59:55.623288 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 07:59:55.623515 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:00.626056 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:00.626237 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:02.219404 2883 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:00:02.219587 2883 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:00:02.220185 2883 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:00:02.221369 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:00:02.223901 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:02.224145 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:00:02.224249 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:00:02.224524 2883 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:00:05.628975 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:05.629184 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:10.631836 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:10.632048 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:15.634789 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:15.634974 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:17.411490 2959 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:00:17.411669 2959 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:00:17.412391 2959 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:00:17.413584 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:00:17.416136 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:17.416342 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:00:17.416421 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:00:17.416612 2959 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260502 08:00:19.785743 1399 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (303 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:19.785753 1260 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (303 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:19.785734 1120 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (303 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 08:00:20.637627 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:20.637825 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:25.640632 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:25.640790 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:28.383117 3010 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:00:28.383355 3010 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:00:28.384135 3010 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:00:28.385388 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:00:28.388060 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:28.388260 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:00:28.388339 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:00:28.388566 3010 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:00:30.643711 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:30.643888 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:35.649057 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:35.649343 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:36.050526 3051 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:00:36.050726 3051 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:00:36.051403 3051 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:00:36.052084 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:00:36.053544 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:36.053758 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:00:36.053869 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:00:36.054158 3051 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:00:40.651993 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:40.652163 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:45.654842 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:45.655009 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:45.876027 3093 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:00:45.876215 3093 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:00:45.876813 3093 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:00:45.877756 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:00:45.879637 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:45.879828 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:00:45.879916 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:00:45.880198 3093 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:00:49.991626 3114 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:00:49.991806 3114 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:00:49.992368 3114 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:00:49.996028 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:00:49.996310 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:00:49.996474 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:00:49.996763 3114 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:00:50.658022 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:50.658248 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:00:55.663278 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:00:55.663513 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:00.666288 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:00.666483 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:05.669268 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:05.669482 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:07.359118 3186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:01:07.359422 3186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:01:07.360805 3186 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:01:07.361899 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 08:01:07.364244 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:07.364475 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:01:07.364571 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:01:07.364853 3186 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:01:10.672212 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:10.672425 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:15.675267 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:15.675472 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:16.553031 3229 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:01:16.553216 3229 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:01:16.553830 3229 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:01:16.554896 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:01:16.556108 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:16.556294 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:01:16.556402 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:01:16.556648 3229 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:01:18.707453 3240 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:01:18.707621 3240 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:01:18.708186 3240 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:01:18.712069 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:18.712304 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:01:18.712404 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:01:18.712671 3240 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260502 08:01:19.931356 1260 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (423 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:19.931493 1120 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (423 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:19.931515 1399 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (423 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 08:01:20.678360 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:20.678565 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:23.941422 3266 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:01:23.941670 3266 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:01:23.942353 3266 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:01:23.943475 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260502 08:01:23.944887 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:23.945103 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:01:23.945194 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:01:23.945454 3266 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:01:25.681372 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:25.681576 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:30.684298 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:30.684458 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:32.765089 3305 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:01:32.765340 3305 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:01:32.766079 3305 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:01:32.768903 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:01:32.772295 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:32.772500 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:01:32.772593 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:01:32.772847 3305 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:01:35.687105 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:35.687278 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:40.689517 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:40.689710 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:40.742389 3340 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:01:40.742641 3340 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:01:40.743362 3340 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:01:40.744062 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:01:40.745656 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:40.745849 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:01:40.745939 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:01:40.746328 3340 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:01:45.692463 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:45.692696 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:50.695813 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:50.696046 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:01:55.337551 3396 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:01:55.337779 3396 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:01:55.338506 3396 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:01:55.339803 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:01:55.344520 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:01:55.344797 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:01:55.344911 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:01:55.345220 3396 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:01:55.698771 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:01:55.699004 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:00.702057 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:00.702350 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:02.962944 3428 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:02:02.963231 3428 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:02:02.963968 3428 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:02:02.965272 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:02:02.968146 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:02:02.968391 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:02:02.968480 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:02:02.968804 3428 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:02:05.705480 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:05.705685 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:10.708827 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:10.709057 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:15.711731 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:15.711898 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:15.856909 3483 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:02:15.857095 3483 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:02:15.857771 3483 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:02:15.858482 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:02:15.859788 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:02:15.859988 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:02:15.860097 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:02:15.860359 3483 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260502 08:02:20.084914 1260 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (543 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:02:20.084996 1120 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (543 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:02:20.085008 1399 heartbeater.cc:646] Failed to heartbeat to 127.0.105.61:33363 (543 consecutive failures): Network error: Failed to ping master at 127.0.105.61:33363: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
I20260502 08:02:20.127663 2253 maintenance_manager.cc:419] P 364807e94fd04d41bfa5976f76f187a6: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 275 bytes on disk
I20260502 08:02:20.132745 2252 maintenance_manager.cc:643] P 364807e94fd04d41bfa5976f76f187a6: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":377,"thread_start_us":487,"threads_started":2,"wal-append.queue_time_us":335}
I20260502 08:02:20.714572 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:20.714730 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:25.717128 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:25.717283 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:30.719964 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:30.720152 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:33.763757 3555 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:02:33.763994 3555 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:02:33.765091 3555 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:02:33.770495 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:02:33.771729 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:02:33.771932 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:02:33.772004 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:02:33.772248 3555 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260502 08:02:35.723860 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:35.724047 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:40.726939 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:40.727171 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260502 08:02:45.438400 3600 raft_consensus.cc:493] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260502 08:02:45.438628 3600 raft_consensus.cc:515] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364807e94fd04d41bfa5976f76f187a6" member_type: VOTER last_known_addr { host: "127.0.105.62" port: 43729 } } peers { permanent_uuid: "51250627cebd4f9abefd40c111865403" member_type: VOTER last_known_addr { host: "127.0.105.61" port: 33363 } } peers { permanent_uuid: "0b49a409ffb84415ad9b53c0a862c470" member_type: VOTER last_known_addr { host: "127.0.105.60" port: 35545 } }
I20260502 08:02:45.439761 3600 leader_election.cc:290] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363), 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545)
W20260502 08:02:45.440836 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260502 08:02:45.442107 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 51250627cebd4f9abefd40c111865403 (127.0.105.61:33363): Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111)
W20260502 08:02:45.442310 2251 leader_election.cc:336] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0b49a409ffb84415ad9b53c0a862c470 (127.0.105.60:35545): Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
I20260502 08:02:45.442412 2251 leader_election.cc:304] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 364807e94fd04d41bfa5976f76f187a6; no voters: 0b49a409ffb84415ad9b53c0a862c470, 51250627cebd4f9abefd40c111865403
I20260502 08:02:45.442675 3600 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 364807e94fd04d41bfa5976f76f187a6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260502 08:02:45.671201 1404 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
W20260502 08:02:45.671520 1404 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
W20260502 08:02:45.671617 1404 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
F20260502 08:02:45.682461 1419 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c61458705db241fd8d267928ef573ae4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1777708965 (unix time) try "date -d @1777708965" if you are using GNU date ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000001a4) received by PID 420 (TID 0x7f3ad23ad700) from PID 420; stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.689963 1411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66ca50c5f9f94d8aa108db59823fb91a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.689963 1411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66ca50c5f9f94d8aa108db59823fb91a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690048 1416 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0fe7ca1b362f4a8e9c78d28d1531545b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.689963 1411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66ca50c5f9f94d8aa108db59823fb91a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690048 1416 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0fe7ca1b362f4a8e9c78d28d1531545b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690212 1432 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.689963 1411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66ca50c5f9f94d8aa108db59823fb91a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690048 1416 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0fe7ca1b362f4a8e9c78d28d1531545b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690212 1432 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690209 1413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_497a847e92ab43d5bd70df89906d8725 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.689963 1411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66ca50c5f9f94d8aa108db59823fb91a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690048 1416 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0fe7ca1b362f4a8e9c78d28d1531545b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690212 1432 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690209 1413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_497a847e92ab43d5bd70df89906d8725 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690407 1435 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f3ae8703980 (unknown) at ??:0
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.689963 1411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66ca50c5f9f94d8aa108db59823fb91a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690048 1416 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0fe7ca1b362f4a8e9c78d28d1531545b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690212 1432 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690209 1413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_497a847e92ab43d5bd70df89906d8725 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690407 1435 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.691915 1415 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b0a63cd3a6fb4f1d86da504a21e8516f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260502 08:02:45.682466 1417 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Fimed out: Error creating table default.Table_97e488bcfc6344bb9ab4a680e6cbc086 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111) 20260502 08:02:45.682466 1418 master-stress-test.cc:297] Check failed: _s.ok() Bad status:
med out: Error creating table default.Table_517dadf96c184083819e9a71e8de9d2f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683754 1420 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d32274c9b0c84d3dadb672c216102926 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.683993 1437 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.685109 1414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd6af681f6ad416ea58e484e70699dfb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.687590 1422 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688669 1433 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.688670 1421 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b3858b7ca5141aebae79d45a08c7f7f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.689963 1411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66ca50c5f9f94d8aa108db59823fb91a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690048 1416 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0fe7ca1b362f4a8e9c78d28d1531545b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690212 1432 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690209 1413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_497a847e92ab43d5bd70df89906d8725 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.690407 1435 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.691915 1415 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b0a63cd3a6fb4f1d86da504a21e8516f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)F20260502 08:02:45.691915 1434 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.105.62:43729,127.0.105.61:33363,127.0.105.60:35545, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.105.60:35545: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f3ae14f3e87 gsignal at ??:0
@ 0x7f3ae14f57f1 abort at ??:0
@ 0x7f3ae422ddcd google::LogMessage::Fail() at ??:0
@ 0x7f3ae4231b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3ae422d7cc google::LogMessage::Flush() at ??:0
@ 0x7f3ae422ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b582 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f3ae1b7b6df (unknown) at ??:0
@ 0x7f3ae86f86db start_thread at ??:0
@ 0x7f3ae15d661f clone at ??:0
I20260502 08:02:45.730260 2232 master_runner.cc:186] Error getting master registration for 127.0.105.61:33363: Network error: Client connection negotiation failed: client connection to 127.0.105.61:33363: connect: Connection refused (error 111),
I20260502 08:02:45.730444 2232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...