Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250904 07:52:54.374658 416 test_util.cc:276] Using random seed: 37828824
2025-09-04T07:52:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T07:52:54Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXge7Va/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXge7Va/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/security.properties
2025-09-04 07:53:00: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXge7Va/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXge7Va/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250904 07:53:08.559949 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34211
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 07:53:08.893263 657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:08.893631 657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:08.893757 657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:08.902354 657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:08.902472 657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:08.902567 657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:08.902649 657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:08.913019 657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:08.914772 657 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:08.916601 657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:08.925557 662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:08.925596 663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:08.927376 665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:08.927909 657 server_base.cc:1047] running on GCE node
I20250904 07:53:08.928755 657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:08.930294 657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:08.931488 657 hybrid_clock.cc:648] HybridClock initialized: now 1756972388931431 us; error 56 us; skew 500 ppm
I20250904 07:53:08.931831 657 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:08.934070 657 webserver.cc:480] Webserver started at http://127.0.104.62:32783/ using document root <none> and password file <none>
I20250904 07:53:08.934811 657 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:08.934916 657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:08.935257 657 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:08.938827 657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data/instance:
uuid: "d6b92e095cf84479a8896513f2acf0a0"
format_stamp: "Formatted at 2025-09-04 07:53:08 on dist-test-slave-jtcb"
I20250904 07:53:08.939644 657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/wal/instance:
uuid: "d6b92e095cf84479a8896513f2acf0a0"
format_stamp: "Formatted at 2025-09-04 07:53:08 on dist-test-slave-jtcb"
I20250904 07:53:08.944762 657 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250904 07:53:08.948004 671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:08.949343 657 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:08.949518 657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/wal
uuid: "d6b92e095cf84479a8896513f2acf0a0"
format_stamp: "Formatted at 2025-09-04 07:53:08 on dist-test-slave-jtcb"
I20250904 07:53:08.949702 657 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:08.985913 657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:08.986855 657 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:08.987143 657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:09.010804 657 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34211
I20250904 07:53:09.010823 722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34211 every 8 connection(s)
I20250904 07:53:09.012282 657 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/data/info.pb
I20250904 07:53:09.016330 723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:09.017536 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 657
I20250904 07:53:09.017859 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-0/wal/instance
I20250904 07:53:09.021934 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 07:53:09.025446 723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } has no permanent_uuid. Determining permanent_uuid...
I20250904 07:53:09.047871 723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } has no permanent_uuid. Determining permanent_uuid...
W20250904 07:53:09.049010 672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
W20250904 07:53:09.050503 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:39659: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:09.099853 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } attempt: 1
W20250904 07:53:09.101920 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:39659: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:09.170214 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } attempt: 2
W20250904 07:53:09.172271 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:39659: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:09.263547 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } attempt: 3
W20250904 07:53:09.265384 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:39659: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
W20250904 07:53:09.360409 724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:09.360774 724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:09.360898 724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:09.369439 724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:09.369554 724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:09.369640 724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:09.369709 724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:09.379256 724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:09.380935 724 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:09.382740 724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:09.391757 733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:09.392048 735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:09.391803 732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:09.392730 724 server_base.cc:1047] running on GCE node
I20250904 07:53:09.393523 724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:09.395080 724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:09.396318 724 hybrid_clock.cc:648] HybridClock initialized: now 1756972389396267 us; error 46 us; skew 500 ppm
I20250904 07:53:09.396667 724 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:09.398867 724 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:09.399583 724 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:09.399688 724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:09.400041 724 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:09.402900 724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/instance:
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:09.403869 724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal/instance:
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:09.408620 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } attempt: 4
I20250904 07:53:09.409698 724 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
W20250904 07:53:09.410398 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:39659: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:09.413249 741 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:09.414626 724 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:09.414801 724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:09.414976 724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:09.429461 724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:09.430871 724 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:09.431131 724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:09.450840 724 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:09.450847 792 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:09.452241 724 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:09.456339 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:09.461262 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 724
I20250904 07:53:09.461424 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal/instance
I20250904 07:53:09.461850 793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } has no permanent_uuid. Determining permanent_uuid...
I20250904 07:53:09.462450 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40359
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 07:53:09.475327 793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } has no permanent_uuid. Determining permanent_uuid...
I20250904 07:53:09.485353 793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
W20250904 07:53:09.486397 743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:53:09.487828 793 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:09.537237 793 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 1
W20250904 07:53:09.539309 793 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:09.607635 793 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 2
W20250904 07:53:09.609735 793 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:09.701067 793 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 3
W20250904 07:53:09.703158 793 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:09.709663 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } attempt: 5
I20250904 07:53:09.716565 723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
W20250904 07:53:09.718488 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:09.769762 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 1
W20250904 07:53:09.771863 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:53:09.809417 795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:09.809743 795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:09.809827 795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:09.818403 795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:09.818507 795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:09.818560 795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:09.818607 795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:09.828315 795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:09.829947 795 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:09.831776 795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250904 07:53:09.840102 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 2
W20250904 07:53:09.840287 803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:09.840934 802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:09.841025 805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:09.842212 723 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:09.842386 795 server_base.cc:1047] running on GCE node
I20250904 07:53:09.843173 795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:09.844537 795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:09.845795 795 hybrid_clock.cc:648] HybridClock initialized: now 1756972389845752 us; error 32 us; skew 500 ppm
I20250904 07:53:09.846186 795 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:09.846387 793 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 4
W20250904 07:53:09.848102 793 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:09.848598 795 webserver.cc:480] Webserver started at http://127.0.104.60:43075/ using document root <none> and password file <none>
I20250904 07:53:09.849339 795 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:09.849432 795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:09.849714 795 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:09.852209 795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/instance:
uuid: "1c32cb6c255941f99f1fd72793331fe5"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:09.852988 795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal/instance:
uuid: "1c32cb6c255941f99f1fd72793331fe5"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:09.859019 795 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250904 07:53:09.862345 811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:09.863760 795 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250904 07:53:09.863932 795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
uuid: "1c32cb6c255941f99f1fd72793331fe5"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:09.864125 795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:09.889271 795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:09.890210 795 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:09.890465 795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:09.911300 795 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:40359
I20250904 07:53:09.911852 862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:40359 every 8 connection(s)
I20250904 07:53:09.912971 795 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
I20250904 07:53:09.916576 863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:09.921801 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 795
I20250904 07:53:09.922051 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal/instance
I20250904 07:53:09.922665 863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } has no permanent_uuid. Determining permanent_uuid...
I20250904 07:53:09.938122 863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } has no permanent_uuid. Determining permanent_uuid...
I20250904 07:53:09.945753 863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
I20250904 07:53:09.948398 723 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 3
I20250904 07:53:09.963362 863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Bootstrap starting.
I20250904 07:53:09.963363 723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: Bootstrap starting.
I20250904 07:53:09.967886 723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:09.967886 863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:09.969810 723 log.cc:826] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:09.969887 863 log.cc:826] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:09.973040 723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: No bootstrap required, opened a new log
I20250904 07:53:09.973037 863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: No bootstrap required, opened a new log
I20250904 07:53:09.980795 863 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:09.980792 723 raft_consensus.cc:357] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:09.981207 863 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:09.981207 723 raft_consensus.cc:383] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:09.981334 863 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32cb6c255941f99f1fd72793331fe5, State: Initialized, Role: FOLLOWER
I20250904 07:53:09.981334 723 raft_consensus.cc:738] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b92e095cf84479a8896513f2acf0a0, State: Initialized, Role: FOLLOWER
I20250904 07:53:09.982221 723 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:09.982220 863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:09.983744 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:09.983748 874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:09.984048 874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:09.984196 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:09.985085 723 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:09.985085 863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: configured and running, proceeding with master startup.
W20250904 07:53:09.989801 677 tablet.cc:2378] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 07:53:10.003144 795 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: OK, instance_id { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" instance_seqno: 1756972389004212 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 07:53:10.011905 795 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:53:10.017560 898 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 07:53:10.022154 898 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 07:53:10.022513 905 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250904 07:53:10.026700 657 master_runner.cc:186] Error getting master registration for 127.0.104.61:39659: OK, instance_id { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" instance_seqno: 1756972389445307 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 07:53:10.033625 657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:53:10.142479 817 tablet.cc:2378] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 07:53:10.147320 793 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } attempt: 5
I20250904 07:53:10.157405 793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:10.161623 793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:10.162740 793 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:10.165110 793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: No bootstrap required, opened a new log
I20250904 07:53:10.170346 793 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:10.170603 793 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:10.170717 793 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:10.171288 793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:10.172247 907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.172519 907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:10.173305 793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
W20250904 07:53:10.183528 747 tablet.cc:2378] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 07:53:10.193329 724 master_runner.cc:263] No leader master found from master 3d34d58088e14ab9ba0e5c9dcb0820e1
I20250904 07:53:10.193471 724 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:10.196542 907 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:10.196805 907 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:10.198503 907 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
I20250904 07:53:10.199689 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b92e095cf84479a8896513f2acf0a0" is_pre_election: true
I20250904 07:53:10.200201 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d34d58088e14ab9ba0e5c9dcb0820e1 in term 0.
W20250904 07:53:10.200513 922 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 07:53:10.200632 922 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 07:53:10.200970 743 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1, d6b92e095cf84479a8896513f2acf0a0; no voters:
I20250904 07:53:10.201385 838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32cb6c255941f99f1fd72793331fe5" is_pre_election: true
I20250904 07:53:10.201499 907 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:10.201643 907 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:10.201749 907 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.201845 838 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d34d58088e14ab9ba0e5c9dcb0820e1 in term 0.
I20250904 07:53:10.205514 907 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:10.206344 907 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 1 election: Requested vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
I20250904 07:53:10.207381 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b92e095cf84479a8896513f2acf0a0"
I20250904 07:53:10.207573 698 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.207947 838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c32cb6c255941f99f1fd72793331fe5"
I20250904 07:53:10.208139 838 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.210907 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d34d58088e14ab9ba0e5c9dcb0820e1 in term 1.
I20250904 07:53:10.211503 743 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1, d6b92e095cf84479a8896513f2acf0a0; no voters:
I20250904 07:53:10.211999 907 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:10.212530 838 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d34d58088e14ab9ba0e5c9dcb0820e1 in term 1.
I20250904 07:53:10.213404 907 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Running, Role: LEADER
I20250904 07:53:10.214049 907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:10.217607 924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d34d58088e14ab9ba0e5c9dcb0820e1. Latest consensus state: current_term: 1 leader_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.217954 924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:10.220708 926 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:10.223078 926 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:10.234010 698 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Refusing update from remote peer 3d34d58088e14ab9ba0e5c9dcb0820e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:10.234423 838 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 1 FOLLOWER]: Refusing update from remote peer 3d34d58088e14ab9ba0e5c9dcb0820e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:10.235241 924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:10.235904 907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:10.261086 874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d34d58088e14ab9ba0e5c9dcb0820e1. Latest consensus state: current_term: 1 leader_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.261332 874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:10.263967 927 mvcc.cc:204] Tried to move back new op lower bound from 7196558910381555712 to 7196558910323294208. Current Snapshot: MvccSnapshot[applied={T|T < 7196558910381555712}]
I20250904 07:53:10.265475 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d34d58088e14ab9ba0e5c9dcb0820e1. Latest consensus state: current_term: 1 leader_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.265704 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:10.269369 907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.269577 907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:10.269814 907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.270025 907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:10.271812 926 catalog_manager.cc:1349] Generated new cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:10.271962 926 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:10.274829 874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.275036 874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:10.286880 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:10.287119 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:10.312146 926 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:10.314080 926 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:10.341548 926 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Generated new TSK 0
I20250904 07:53:10.342854 926 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:10.344259 926 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 07:53:10.346102 926 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 07:53:10.352707 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250904 07:53:10.771806 936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:10.772143 936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:10.772331 936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:10.780639 936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:10.780869 936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250904 07:53:10.791430 936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:10.793356 936 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:10.795183 936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:10.804425 942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:10.804705 941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:10.805079 945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:10.805665 936 server_base.cc:1047] running on GCE node
I20250904 07:53:10.806274 936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:10.807165 936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:10.808372 936 hybrid_clock.cc:648] HybridClock initialized: now 1756972390808332 us; error 35 us; skew 500 ppm
I20250904 07:53:10.808671 936 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:10.811305 936 webserver.cc:480] Webserver started at http://127.0.104.1:45553/ using document root <none> and password file <none>
I20250904 07:53:10.812034 936 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:10.812139 936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:10.812461 936 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:10.814917 936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data/instance:
uuid: "2fc1b8ca505440228c8e9a544e3c549a"
format_stamp: "Formatted at 2025-09-04 07:53:10 on dist-test-slave-jtcb"
I20250904 07:53:10.815677 936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/wal/instance:
uuid: "2fc1b8ca505440228c8e9a544e3c549a"
format_stamp: "Formatted at 2025-09-04 07:53:10 on dist-test-slave-jtcb"
I20250904 07:53:10.820879 936 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250904 07:53:10.824141 950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:10.825392 936 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:10.825534 936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/wal
uuid: "2fc1b8ca505440228c8e9a544e3c549a"
format_stamp: "Formatted at 2025-09-04 07:53:10 on dist-test-slave-jtcb"
I20250904 07:53:10.825711 936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:10.842059 955 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d34d58088e14ab9ba0e5c9dcb0820e1)
I20250904 07:53:10.842254 955 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:10.843590 955 leader_election.cc:290] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
I20250904 07:53:10.844177 838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b92e095cf84479a8896513f2acf0a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c32cb6c255941f99f1fd72793331fe5" is_pre_election: true
I20250904 07:53:10.844195 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b92e095cf84479a8896513f2acf0a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" is_pre_election: true
I20250904 07:53:10.844847 672 leader_election.cc:304] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b92e095cf84479a8896513f2acf0a0; no voters: 1c32cb6c255941f99f1fd72793331fe5, 3d34d58088e14ab9ba0e5c9dcb0820e1
I20250904 07:53:10.845191 955 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 07:53:10.868172 936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:10.869165 936 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:10.869472 936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:10.870530 936 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:10.872799 936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:10.872900 936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:10.872992 936 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:10.873051 936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:10.911245 936 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:41509
I20250904 07:53:10.911345 1063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:41509 every 8 connection(s)
I20250904 07:53:10.912662 936 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/data/info.pb
I20250904 07:53:10.921456 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 936
I20250904 07:53:10.921706 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-0/wal/instance
I20250904 07:53:10.929718 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 07:53:10.956001 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:10.956454 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:10.957607 1066 heartbeater.cc:507] Master 127.0.104.61:39659 requested a full tablet report, sending...
I20250904 07:53:10.958300 1065 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34211
I20250904 07:53:10.958464 1065 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:10.959543 1065 heartbeater.cc:507] Master 127.0.104.62:34211 requested a full tablet report, sending...
I20250904 07:53:10.960115 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:10.960215 1064 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:10.960412 758 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:10.960614 1064 heartbeater.cc:507] Master 127.0.104.60:40359 requested a full tablet report, sending...
I20250904 07:53:10.962141 688 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:10.962615 758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:44351
I20250904 07:53:10.964423 828 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:11.025126 898 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:11.025261 898 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: loading cluster ID for follower catalog manager: success
I20250904 07:53:11.029944 898 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: acquiring CA information for follower catalog manager: success
I20250904 07:53:11.033339 898 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:11.036651 905 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:11.036844 905 catalog_manager.cc:1554] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: loading cluster ID for follower catalog manager: success
I20250904 07:53:11.041046 905 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: acquiring CA information for follower catalog manager: success
I20250904 07:53:11.043540 905 catalog_manager.cc:1604] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250904 07:53:11.314333 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:11.314682 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:11.314811 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:11.324244 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:11.324473 1072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250904 07:53:11.335246 1072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:11.336998 1072 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:11.339025 1072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:11.349191 1080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:11.349462 1081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:11.350385 1083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:11.351725 1072 server_base.cc:1047] running on GCE node
I20250904 07:53:11.352458 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:11.353650 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:11.354900 1072 hybrid_clock.cc:648] HybridClock initialized: now 1756972391354838 us; error 54 us; skew 500 ppm
I20250904 07:53:11.355298 1072 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:11.358232 1072 webserver.cc:480] Webserver started at http://127.0.104.2:38461/ using document root <none> and password file <none>
I20250904 07:53:11.359359 1072 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:11.359500 1072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:11.360069 1072 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:11.363613 1072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data/instance:
uuid: "45f0af8a16f14b8b83950e205831688f"
format_stamp: "Formatted at 2025-09-04 07:53:11 on dist-test-slave-jtcb"
I20250904 07:53:11.364689 1072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/wal/instance:
uuid: "45f0af8a16f14b8b83950e205831688f"
format_stamp: "Formatted at 2025-09-04 07:53:11 on dist-test-slave-jtcb"
I20250904 07:53:11.371682 1072 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250904 07:53:11.376205 1089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:11.377841 1072 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:11.378079 1072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/wal
uuid: "45f0af8a16f14b8b83950e205831688f"
format_stamp: "Formatted at 2025-09-04 07:53:11 on dist-test-slave-jtcb"
I20250904 07:53:11.378324 1072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:11.397209 1072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:11.398205 1072 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:11.398491 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:11.399466 1072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:11.401114 1072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:11.401216 1072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:11.401329 1072 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:11.401400 1072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:11.445263 1072 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44461
I20250904 07:53:11.445346 1201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44461 every 8 connection(s)
I20250904 07:53:11.446743 1072 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/data/info.pb
I20250904 07:53:11.448768 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 1072
I20250904 07:53:11.449057 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-1/wal/instance
I20250904 07:53:11.456012 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 07:53:11.465610 1066 heartbeater.cc:499] Master 127.0.104.61:39659 was elected leader, sending a full tablet report...
I20250904 07:53:11.482919 1202 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:11.483464 1202 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:11.484879 1202 heartbeater.cc:507] Master 127.0.104.60:40359 requested a full tablet report, sending...
I20250904 07:53:11.487555 828 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:11.489387 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:11.489967 1203 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34211
I20250904 07:53:11.490099 1204 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:11.490161 1203 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:11.490700 1204 heartbeater.cc:507] Master 127.0.104.61:39659 requested a full tablet report, sending...
I20250904 07:53:11.490741 1203 heartbeater.cc:507] Master 127.0.104.62:34211 requested a full tablet report, sending...
I20250904 07:53:11.491644 758 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:11.491674 688 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:11.492652 758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:44835
W20250904 07:53:11.808220 1206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:11.808526 1206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:11.808626 1206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:11.817000 1206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:11.817191 1206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250904 07:53:11.827090 1206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:11.828636 1206 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:11.830430 1206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:11.841420 1216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:11.841421 1217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:11.841464 1219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:11.842062 1206 server_base.cc:1047] running on GCE node
I20250904 07:53:11.842653 1206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:11.843506 1206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:11.844703 1206 hybrid_clock.cc:648] HybridClock initialized: now 1756972391844643 us; error 59 us; skew 500 ppm
I20250904 07:53:11.845044 1206 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:11.847177 1206 webserver.cc:480] Webserver started at http://127.0.104.3:33139/ using document root <none> and password file <none>
I20250904 07:53:11.847916 1206 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:11.848013 1206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:11.848320 1206 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:11.850668 1206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data/instance:
uuid: "7fce49d1a43d4cdbbfde7f0692eee50b"
format_stamp: "Formatted at 2025-09-04 07:53:11 on dist-test-slave-jtcb"
I20250904 07:53:11.851387 1206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal/instance:
uuid: "7fce49d1a43d4cdbbfde7f0692eee50b"
format_stamp: "Formatted at 2025-09-04 07:53:11 on dist-test-slave-jtcb"
I20250904 07:53:11.857095 1206 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20250904 07:53:11.860549 1225 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:11.861829 1206 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:11.862116 1206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal
uuid: "7fce49d1a43d4cdbbfde7f0692eee50b"
format_stamp: "Formatted at 2025-09-04 07:53:11 on dist-test-slave-jtcb"
I20250904 07:53:11.862337 1206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:11.882705 1206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:11.883711 1206 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:11.884016 1206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:11.885020 1206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:11.887312 1206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:11.887411 1206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:11.887526 1206 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:11.887598 1206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:11.924886 1206 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44879
I20250904 07:53:11.924934 1337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44879 every 8 connection(s)
I20250904 07:53:11.926627 1206 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/data/info.pb
I20250904 07:53:11.928396 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 1206
I20250904 07:53:11.928586 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal/instance
I20250904 07:53:11.947788 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:11.947824 1338 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:11.948257 1339 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34211
I20250904 07:53:11.948215 1338 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:11.948216 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:11.948400 1339 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:11.949456 1338 heartbeater.cc:507] Master 127.0.104.60:40359 requested a full tablet report, sending...
I20250904 07:53:11.949417 1339 heartbeater.cc:507] Master 127.0.104.62:34211 requested a full tablet report, sending...
I20250904 07:53:11.949417 1340 heartbeater.cc:507] Master 127.0.104.61:39659 requested a full tablet report, sending...
I20250904 07:53:11.951174 758 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:11.951326 688 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:11.951372 827 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:11.952162 758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:57327
I20250904 07:53:11.956980 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 07:53:11.994935 1204 heartbeater.cc:499] Master 127.0.104.61:39659 was elected leader, sending a full tablet report...
I20250904 07:53:11.995241 755 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_aa64673a1d5e47bdb0b862084daac8ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.992022 756 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_406165ecf1be41718f1b7c6d718a1d55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.995697 754 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_da1c79fbee874fd58c1a2dfea10b89c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.996066 753 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_2fcb873f751947deb8f788e89007069c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.996474 750 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_5d14da275d34445abdb109af99d1cc7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.997666 751 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_c308554984c24f29b70e853b6e48516b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.997952 749 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_dcff30abce7b454b827a08daf85eab8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.991474 757 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_5ff476a678d047848753ef2b12175785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.995903 752 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_28be2f0ae2d54f5bb0b09107f6069cc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:11.991420 758 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56674:
name: "default.Table_ad93fa23e5724bf3aaa93e872f55d612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:12.011605 755 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aa64673a1d5e47bdb0b862084daac8ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:12.051514 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXge7Va/build/asan/bin/kudu pid 724
I20250904 07:53:12.053067 788 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 07:53:12.200743 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXge7Va/build/asan/bin/kudu with pid 724
W20250904 07:53:12.233477 1345 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:39659)
I20250904 07:53:12.235718 1385 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d34d58088e14ab9ba0e5c9dcb0820e1)
I20250904 07:53:12.235957 1385 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:12.236752 1385 leader_election.cc:290] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
I20250904 07:53:12.237819 838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b92e095cf84479a8896513f2acf0a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c32cb6c255941f99f1fd72793331fe5" is_pre_election: true
W20250904 07:53:12.239981 672 connection.cc:537] client connection to 127.0.104.61:39659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:12.241907 1351 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:12.234992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:39659 (local address 127.0.0.1:56688)
0904 07:53:12.235364 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0904 07:53:12.235389 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0904 07:53:12.235584 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 07:53:12.240315 (+ 4731us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":105}
I20250904 07:53:12.246412 1387 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d34d58088e14ab9ba0e5c9dcb0820e1)
I20250904 07:53:12.246697 1387 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:12.249662 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c32cb6c255941f99f1fd72793331fe5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b92e095cf84479a8896513f2acf0a0" is_pre_election: true
I20250904 07:53:12.249915 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c32cb6c255941f99f1fd72793331fe5 in term 1.
I20250904 07:53:12.250499 813 leader_election.cc:304] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0; no voters:
I20250904 07:53:12.262421 1388 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 07:53:12.262749 1388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d34d58088e14ab9ba0e5c9dcb0820e1)
I20250904 07:53:12.263031 1388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:12.263159 1387 leader_election.cc:290] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659)
I20250904 07:53:12.266474 1388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
W20250904 07:53:12.267113 812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:12.267356 1388 leader_election.cc:290] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 2 election: Requested vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659)
I20250904 07:53:12.269831 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c32cb6c255941f99f1fd72793331fe5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b92e095cf84479a8896513f2acf0a0"
I20250904 07:53:12.270063 698 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:12.272886 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c32cb6c255941f99f1fd72793331fe5 in term 2.
I20250904 07:53:12.273455 813 leader_election.cc:304] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0; no voters:
I20250904 07:53:12.274083 1388 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 07:53:12.274353 1388 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 LEADER]: Becoming Leader. State: Replica: 1c32cb6c255941f99f1fd72793331fe5, State: Running, Role: LEADER
I20250904 07:53:12.274899 1388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
W20250904 07:53:12.285243 812 leader_election.cc:336] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659): Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:12.286374 1387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c32cb6c255941f99f1fd72793331fe5. Latest consensus state: current_term: 2 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:12.286758 1387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: LEADER
W20250904 07:53:12.288084 812 leader_election.cc:336] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659): Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:12.289011 1393 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
W20250904 07:53:12.289831 672 leader_election.cc:336] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659): Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:12.291626 1393 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:12.290100 672 leader_election.cc:304] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b92e095cf84479a8896513f2acf0a0; no voters: 1c32cb6c255941f99f1fd72793331fe5, 3d34d58088e14ab9ba0e5c9dcb0820e1
I20250904 07:53:12.291950 1385 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 07:53:12.292687 1393 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:12.292836 1393 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:12.294245 1393 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:12.295131 1393 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Loaded TSK: 0
I20250904 07:53:12.296308 1393 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:12.299015 1393 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 07:53:12.299969 1393 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 07:53:12.311205 827 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_dcff30abce7b454b827a08daf85eab8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.312124 822 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_406165ecf1be41718f1b7c6d718a1d55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.312188 825 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_28be2f0ae2d54f5bb0b09107f6069cc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.311544 824 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_5ff476a678d047848753ef2b12175785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.312834 823 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_ad93fa23e5724bf3aaa93e872f55d612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.312983 820 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_da1c79fbee874fd58c1a2dfea10b89c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.313544 821 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_aa64673a1d5e47bdb0b862084daac8ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.311643 828 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_c308554984c24f29b70e853b6e48516b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.314168 819 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_2fcb873f751947deb8f788e89007069c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:12.311252 826 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "default.Table_5d14da275d34445abdb109af99d1cc7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:12.323972 822 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_406165ecf1be41718f1b7c6d718a1d55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.326382 824 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5ff476a678d047848753ef2b12175785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:12.454478 1340 heartbeater.cc:499] Master 127.0.104.61:39659 was elected leader, sending a full tablet report...
W20250904 07:53:12.460544 1340 heartbeater.cc:646] Failed to heartbeat to 127.0.104.61:39659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
W20250904 07:53:12.476552 1066 heartbeater.cc:646] Failed to heartbeat to 127.0.104.61:39659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
W20250904 07:53:12.502393 1204 heartbeater.cc:646] Failed to heartbeat to 127.0.104.61:39659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
I20250904 07:53:12.549458 824 catalog_manager.cc:2149] created HMS catalog entry for table [id=783aa6a2ef4a497193e19a03874a4b60]
I20250904 07:53:12.555016 698 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Refusing update from remote peer 1c32cb6c255941f99f1fd72793331fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20250904 07:53:12.555575 812 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 -> Peer 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659): Couldn't send request to peer 3d34d58088e14ab9ba0e5c9dcb0820e1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:12.555773 1387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250904 07:53:12.558538 1385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c32cb6c255941f99f1fd72793331fe5. Latest consensus state: current_term: 2 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:12.558753 1385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:12.566196 1398 mvcc.cc:204] Tried to move back new op lower bound from 7196558919892676608 to 7196558918762958848. Current Snapshot: MvccSnapshot[applied={T|T < 7196558919892676608}]
I20250904 07:53:12.566844 1397 mvcc.cc:204] Tried to move back new op lower bound from 7196558919892676608 to 7196558918762958848. Current Snapshot: MvccSnapshot[applied={T|T < 7196558919892676608}]
I20250904 07:53:12.567269 1387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:12.567497 1387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:12.568183 1385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:12.568388 1385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:12.572822 824 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36264:
table { table_name: "default.Table_5ff476a678d047848753ef2b12175785" } new_table_name: "default.Table_eea7810b592e4f46aea67419484a1c36" modify_external_catalogs: true
I20250904 07:53:12.596851 822 catalog_manager.cc:2149] created HMS catalog entry for table [id=71f73db7e1dc4d9398193a557ca8d3a9]
W20250904 07:53:12.602430 823 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ad93fa23e5724bf3aaa93e872f55d612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.602815 828 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c308554984c24f29b70e853b6e48516b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.603732 820 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da1c79fbee874fd58c1a2dfea10b89c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.603739 819 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2fcb873f751947deb8f788e89007069c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.604033 825 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_28be2f0ae2d54f5bb0b09107f6069cc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.602468 821 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aa64673a1d5e47bdb0b862084daac8ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.605098 827 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dcff30abce7b454b827a08daf85eab8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250904 07:53:12.606105 826 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5d14da275d34445abdb109af99d1cc7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:12.621184 822 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36264:
table { table_name: "default.Table_406165ecf1be41718f1b7c6d718a1d55" } new_table_name: "default.Table_67f27d59debb40fda234fcef2ef93524" modify_external_catalogs: true
F20250904 07:53:12.628255 897 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 3 received events: NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_5ff476a678d047848753ef2b12175785, message={"server":"","servicePrincipal":"","db":"default","table":"table_5ff476a678d047848753ef2b12175785","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5ff476a678d047848753ef2b12175785\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_5ff476a678d047848753ef2b12175785\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5ff476a678d047848753ef2b12175785\",\"kudu.table_id\":\"783aa6a2ef4a497193e19a03874a4b60\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_aa64673a1d5e47bdb0b862084daac8ca, message={"server":"","servicePrincipal":"","db":"default","table":"table_aa64673a1d5e47bdb0b862084daac8ca","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_aa64673a1d5e47bdb0b862084daac8ca\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_aa64673a1d5e47bdb0b862084daac8ca\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_aa64673a1d5e47bdb0b862084daac8ca\",\"kudu.table_id\":\"02e00655160b41698e3454c7104042bc\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7ffaae20ddcd google::LogMessage::Fail() at ??:0
I20250904 07:53:12.631138 1272 tablet_service.cc:1468] Processing CreateTablet for tablet 65c4e146fd534eb2bccc6867be1e755d (DEFAULT_TABLE table=default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:12.630733 1273 tablet_service.cc:1468] Processing CreateTablet for tablet d1bc320027bd43b59a7fc77136aeda74 (DEFAULT_TABLE table=default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7ffaae211b93 google::LogMessage::SendToLog() at ??:0
I20250904 07:53:12.632593 1272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c4e146fd534eb2bccc6867be1e755d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.633083 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bc320027bd43b59a7fc77136aeda74. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ffaae20d7cc google::LogMessage::Flush() at ??:0
@ 0x7ffaae20da11 google::LogMessage::~LogMessage() at ??:0
I20250904 07:53:12.643975 1136 tablet_service.cc:1468] Processing CreateTablet for tablet 65c4e146fd534eb2bccc6867be1e755d (DEFAULT_TABLE table=default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:12.645572 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c4e146fd534eb2bccc6867be1e755d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.646462 999 tablet_service.cc:1468] Processing CreateTablet for tablet d1bc320027bd43b59a7fc77136aeda74 (DEFAULT_TABLE table=default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:12.647929 999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bc320027bd43b59a7fc77136aeda74. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ffac46507b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 07:53:12.650358 1137 tablet_service.cc:1468] Processing CreateTablet for tablet d1bc320027bd43b59a7fc77136aeda74 (DEFAULT_TABLE table=default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:12.650981 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bc320027bd43b59a7fc77136aeda74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.646358 998 tablet_service.cc:1468] Processing CreateTablet for tablet 65c4e146fd534eb2bccc6867be1e755d (DEFAULT_TABLE table=default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:12.654563 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c4e146fd534eb2bccc6867be1e755d. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ffac464f061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250904 07:53:12.658504 1409 tablet_bootstrap.cc:492] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b: Bootstrap starting.
@ 0x7ffac4656cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250904 07:53:12.664974 1409 tablet_bootstrap.cc:654] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.666365 1409 log.cc:826] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:12.672739 1409 tablet_bootstrap.cc:492] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b: No bootstrap required, opened a new log
I20250904 07:53:12.675933 1409 ts_tablet_manager.cc:1397] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
@ 0x7ffac4656b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7ffab0f762b9 kudu::Thread::SuperviseThread() at ??:0
I20250904 07:53:12.695552 1412 tablet_bootstrap.cc:492] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a: Bootstrap starting.
I20250904 07:53:12.697125 1409 raft_consensus.cc:357] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:12.697540 1409 raft_consensus.cc:383] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.697661 1409 raft_consensus.cc:738] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fce49d1a43d4cdbbfde7f0692eee50b, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.698392 1409 consensus_queue.cc:260] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
@ 0x7ffab439a6db start_thread at ??:0
I20250904 07:53:12.699819 1412 tablet_bootstrap.cc:654] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a: Neither blocks nor log segments found. Creating new log.
@ 0x7ffaad0be71f clone at ??:0
I20250904 07:53:12.702263 1415 tablet_bootstrap.cc:492] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f: Bootstrap starting.
I20250904 07:53:12.705879 1415 tablet_bootstrap.cc:654] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.707794 1415 log.cc:826] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:12.709219 1412 log.cc:826] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:12.709761 1409 ts_tablet_manager.cc:1428] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b: Time spent starting tablet: real 0.034s user 0.013s sys 0.003s
I20250904 07:53:12.709890 1338 heartbeater.cc:499] Master 127.0.104.60:40359 was elected leader, sending a full tablet report...
I20250904 07:53:12.711334 1409 tablet_bootstrap.cc:492] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: Bootstrap starting.
I20250904 07:53:12.713109 1409 tablet_bootstrap.cc:654] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.724004 1415 tablet_bootstrap.cc:492] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f: No bootstrap required, opened a new log
I20250904 07:53:12.724336 1415 ts_tablet_manager.cc:1397] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.003s
I20250904 07:53:12.724795 1409 tablet_bootstrap.cc:492] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: No bootstrap required, opened a new log
I20250904 07:53:12.724956 1409 ts_tablet_manager.cc:1397] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250904 07:53:12.726939 1409 raft_consensus.cc:357] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:12.728788 1409 raft_consensus.cc:383] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.728884 1409 raft_consensus.cc:738] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fce49d1a43d4cdbbfde7f0692eee50b, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.729223 1409 consensus_queue.cc:260] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:12.729738 1409 ts_tablet_manager.cc:1428] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250904 07:53:12.734814 1412 tablet_bootstrap.cc:492] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a: No bootstrap required, opened a new log
I20250904 07:53:12.735154 1412 ts_tablet_manager.cc:1397] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.000s
I20250904 07:53:12.744133 1412 raft_consensus.cc:357] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:12.744604 1412 raft_consensus.cc:383] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.744730 1412 raft_consensus.cc:738] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc1b8ca505440228c8e9a544e3c549a, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.748346 1415 raft_consensus.cc:357] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:12.748790 1415 raft_consensus.cc:383] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.748909 1415 raft_consensus.cc:738] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0af8a16f14b8b83950e205831688f, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.749785 1415 consensus_queue.cc:260] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:12.750653 1412 consensus_queue.cc:260] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:12.750847 1202 heartbeater.cc:499] Master 127.0.104.60:40359 was elected leader, sending a full tablet report...
I20250904 07:53:12.752285 1415 ts_tablet_manager.cc:1428] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f: Time spent starting tablet: real 0.028s user 0.009s sys 0.003s
I20250904 07:53:12.757309 1415 tablet_bootstrap.cc:492] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f: Bootstrap starting.
I20250904 07:53:12.758704 1064 heartbeater.cc:499] Master 127.0.104.60:40359 was elected leader, sending a full tablet report...
I20250904 07:53:12.759153 1415 tablet_bootstrap.cc:654] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.761597 1412 ts_tablet_manager.cc:1428] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a: Time spent starting tablet: real 0.026s user 0.011s sys 0.000s
I20250904 07:53:12.763948 1412 tablet_bootstrap.cc:492] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a: Bootstrap starting.
I20250904 07:53:12.765813 1412 tablet_bootstrap.cc:654] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.771733 1415 tablet_bootstrap.cc:492] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f: No bootstrap required, opened a new log
I20250904 07:53:12.772208 1415 ts_tablet_manager.cc:1397] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250904 07:53:12.773844 1412 tablet_bootstrap.cc:492] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a: No bootstrap required, opened a new log
I20250904 07:53:12.773828 1415 raft_consensus.cc:357] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:12.774051 1412 ts_tablet_manager.cc:1397] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20250904 07:53:12.774058 1415 raft_consensus.cc:383] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.774127 1415 raft_consensus.cc:738] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0af8a16f14b8b83950e205831688f, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.774425 1415 consensus_queue.cc:260] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:12.774896 1415 ts_tablet_manager.cc:1428] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:12.776273 1412 raft_consensus.cc:357] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:12.776473 1412 raft_consensus.cc:383] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.776537 1412 raft_consensus.cc:738] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc1b8ca505440228c8e9a544e3c549a, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.777426 1412 consensus_queue.cc:260] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:12.777956 1412 ts_tablet_manager.cc:1428] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 07:53:12.824030 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=45475
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 07:53:12.923281 1067 tablet.cc:2378] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 07:53:12.935570 1341 tablet.cc:2378] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 07:53:12.945178 1227 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:12.945267 1346 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:12.945747 1091 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:12.947258 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.60:40359)
W20250904 07:53:12.947363 952 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:12.947733 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.60:40359)
W20250904 07:53:12.948486 1091 connection.cc:537] server connection from 127.0.0.1:58088 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:12.950690 1394 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:12.946530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:40359 (local address 127.0.104.3:46227)
0904 07:53:12.946787 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0904 07:53:12.946806 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0904 07:53:12.946946 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 07:53:12.949855 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150}
W20250904 07:53:12.950713 1396 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:12.946622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:40359 (local address 127.0.104.2:58001)
0904 07:53:12.946849 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0904 07:53:12.946867 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0904 07:53:12.946988 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 07:53:12.949852 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126}
W20250904 07:53:12.953440 1395 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:12.948193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:40359 (local address 127.0.104.1:46971)
0904 07:53:12.948358 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0904 07:53:12.948374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0904 07:53:12.948482 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0904 07:53:12.952457 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":43}
W20250904 07:53:12.984583 1211 tablet.cc:2378] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 07:53:13.127912 1435 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:13.128438 1435 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:13.130038 1435 leader_election.cc:290] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:13.131505 672 leader_election.cc:336] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659): Network error: Client connection negotiation failed: client connection to 127.0.104.61:39659: connect: Connection refused (error 111)
W20250904 07:53:13.131660 673 leader_election.cc:336] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:13.131759 673 leader_election.cc:304] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b92e095cf84479a8896513f2acf0a0; no voters: 1c32cb6c255941f99f1fd72793331fe5, 3d34d58088e14ab9ba0e5c9dcb0820e1
I20250904 07:53:13.132134 1435 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250904 07:53:13.168890 1432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:13.169232 1432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:13.169350 1432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:13.177553 1432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:13.177671 1432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:13.177771 1432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:13.177842 1432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:13.187781 1432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:13.189443 1432 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:13.191632 1432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:13.199703 1444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:13.199586 1446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:13.199748 1443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:13.200388 1432 server_base.cc:1047] running on GCE node
I20250904 07:53:13.201097 1432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:13.202458 1432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:13.203683 1432 hybrid_clock.cc:648] HybridClock initialized: now 1756972393203625 us; error 45 us; skew 500 ppm
I20250904 07:53:13.204011 1432 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:13.206255 1432 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:13.206943 1432 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:13.207041 1432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:13.212340 1432 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20250904 07:53:13.215504 1452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:13.216902 1432 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250904 07:53:13.217057 1432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:13.217720 1432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:13.234081 1432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:13.235136 1432 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:13.235430 1432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:13.254140 1432 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:13.254256 1503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:13.255868 1432 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:13.262452 1504 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:13.262907 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 1432
I20250904 07:53:13.267501 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:13.278674 1504 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:13.285356 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:13.288622 1510 raft_consensus.cc:491] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:13.288990 1510 raft_consensus.cc:513] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:13.290493 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:13.290920 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:13.291574 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap complete.
I20250904 07:53:13.292749 1510 leader_election.cc:290] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:13.298585 1504 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:13.299379 1504 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:13.299438 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65c4e146fd534eb2bccc6867be1e755d" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0af8a16f14b8b83950e205831688f" is_pre_election: true
I20250904 07:53:13.299737 1293 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65c4e146fd534eb2bccc6867be1e755d" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" is_pre_election: true
I20250904 07:53:13.299896 1157 raft_consensus.cc:2466] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 0.
I20250904 07:53:13.300127 1293 raft_consensus.cc:2466] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 0.
I20250904 07:53:13.300174 1504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:13.300526 953 leader_election.cc:304] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f; no voters:
I20250904 07:53:13.300908 1510 raft_consensus.cc:2802] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:13.301050 1510 raft_consensus.cc:491] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:13.301141 1510 raft_consensus.cc:3058] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.301038 1514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:13.301314 1514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:13.302176 1504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:13.303186 1510 raft_consensus.cc:513] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:13.303838 1510 leader_election.cc:290] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 election: Requested vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:13.304315 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65c4e146fd534eb2bccc6867be1e755d" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0af8a16f14b8b83950e205831688f"
I20250904 07:53:13.304528 1293 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65c4e146fd534eb2bccc6867be1e755d" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b"
I20250904 07:53:13.304529 1157 raft_consensus.cc:3058] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.304711 1293 raft_consensus.cc:3058] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.307636 1293 raft_consensus.cc:2466] T 65c4e146fd534eb2bccc6867be1e755d P 7fce49d1a43d4cdbbfde7f0692eee50b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 1.
I20250904 07:53:13.308064 951 leader_election.cc:304] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1b8ca505440228c8e9a544e3c549a, 7fce49d1a43d4cdbbfde7f0692eee50b; no voters:
I20250904 07:53:13.308276 1157 raft_consensus.cc:2466] T 65c4e146fd534eb2bccc6867be1e755d P 45f0af8a16f14b8b83950e205831688f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 1.
I20250904 07:53:13.308531 1510 raft_consensus.cc:2802] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:13.308955 1510 raft_consensus.cc:695] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [term 1 LEADER]: Becoming Leader. State: Replica: 2fc1b8ca505440228c8e9a544e3c549a, State: Running, Role: LEADER
I20250904 07:53:13.309468 1510 consensus_queue.cc:237] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } }
I20250904 07:53:13.310155 1522 raft_consensus.cc:491] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:13.310322 1522 raft_consensus.cc:513] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:13.311489 1293 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bc320027bd43b59a7fc77136aeda74" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" is_pre_election: true
I20250904 07:53:13.311756 1293 raft_consensus.cc:2466] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 0.
I20250904 07:53:13.311988 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bc320027bd43b59a7fc77136aeda74" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0af8a16f14b8b83950e205831688f" is_pre_election: true
I20250904 07:53:13.312238 1157 raft_consensus.cc:2466] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 0.
I20250904 07:53:13.312451 951 leader_election.cc:304] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1b8ca505440228c8e9a544e3c549a, 7fce49d1a43d4cdbbfde7f0692eee50b; no voters:
I20250904 07:53:13.312933 1522 leader_election.cc:290] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879), 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:13.313246 1522 raft_consensus.cc:2802] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:13.313417 1522 raft_consensus.cc:491] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:13.313585 1522 raft_consensus.cc:3058] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.317581 1522 raft_consensus.cc:513] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:13.318997 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bc320027bd43b59a7fc77136aeda74" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0af8a16f14b8b83950e205831688f"
I20250904 07:53:13.319235 1157 raft_consensus.cc:3058] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.322754 1157 raft_consensus.cc:2466] T d1bc320027bd43b59a7fc77136aeda74 P 45f0af8a16f14b8b83950e205831688f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 1.
I20250904 07:53:13.323386 953 leader_election.cc:304] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f; no voters:
I20250904 07:53:13.323532 1293 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bc320027bd43b59a7fc77136aeda74" candidate_uuid: "2fc1b8ca505440228c8e9a544e3c549a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b"
I20250904 07:53:13.323784 1293 raft_consensus.cc:3058] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.323845 1522 leader_election.cc:290] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [CANDIDATE]: Term 1 election: Requested vote from peers 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879), 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:13.324158 1522 raft_consensus.cc:2802] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:13.324370 1522 raft_consensus.cc:695] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [term 1 LEADER]: Becoming Leader. State: Replica: 2fc1b8ca505440228c8e9a544e3c549a, State: Running, Role: LEADER
I20250904 07:53:13.324774 1522 consensus_queue.cc:237] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } } peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } }
I20250904 07:53:13.329591 1293 raft_consensus.cc:2466] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1b8ca505440228c8e9a544e3c549a in term 1.
I20250904 07:53:13.344977 1532 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:13.345111 1532 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: success
I20250904 07:53:13.348867 1532 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: acquiring CA information for follower catalog manager: success
I20250904 07:53:13.351269 1532 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:13.367837 1432 master_runner.cc:186] Error getting master registration for 127.0.104.60:40359: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111),
I20250904 07:53:13.371280 1432 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:13.721628 1537 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:13.721854 1537 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:13.722623 1537 leader_election.cc:290] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:13.725000 673 leader_election.cc:336] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:13.729002 1479 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b92e095cf84479a8896513f2acf0a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" is_pre_election: true
I20250904 07:53:13.729489 1479 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b92e095cf84479a8896513f2acf0a0 in term 1.
I20250904 07:53:13.729924 672 leader_election.cc:304] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1, d6b92e095cf84479a8896513f2acf0a0; no voters: 1c32cb6c255941f99f1fd72793331fe5
I20250904 07:53:13.730224 1537 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 07:53:13.730340 1537 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:13.730410 1537 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:13.732353 1537 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:13.732874 1537 leader_election.cc:290] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
I20250904 07:53:13.733217 1479 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b92e095cf84479a8896513f2acf0a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
I20250904 07:53:13.733398 1479 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 1 FOLLOWER]: Advancing to term 3
W20250904 07:53:13.734036 673 leader_election.cc:336] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:13.738409 1479 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b92e095cf84479a8896513f2acf0a0 in term 3.
I20250904 07:53:13.738862 672 leader_election.cc:304] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1, d6b92e095cf84479a8896513f2acf0a0; no voters: 1c32cb6c255941f99f1fd72793331fe5
I20250904 07:53:13.739161 1537 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 3 FOLLOWER]: Leader election won for term 3
I20250904 07:53:13.739606 1537 raft_consensus.cc:695] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [term 3 LEADER]: Becoming Leader. State: Replica: d6b92e095cf84479a8896513f2acf0a0, State: Running, Role: LEADER
I20250904 07:53:13.740160 1537 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:13.741585 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:13.741768 1340 heartbeater.cc:499] Master 127.0.104.61:39659 was elected leader, sending a full tablet report...
I20250904 07:53:13.741950 1541 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b92e095cf84479a8896513f2acf0a0. Latest consensus state: current_term: 3 leader_uuid: "d6b92e095cf84479a8896513f2acf0a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:13.742255 1541 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:13.742911 1543 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:13.743984 1469 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:52467; Asking this server to re-register.
I20250904 07:53:13.744592 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:13.745671 1543 catalog_manager.cc:671] Loaded metadata for table default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9]
I20250904 07:53:13.746099 1469 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:13.746241 1543 catalog_manager.cc:671] Loaded metadata for table default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60]
I20250904 07:53:13.747874 1543 tablet_loader.cc:96] loaded metadata for tablet 65c4e146fd534eb2bccc6867be1e755d (table default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9])
I20250904 07:53:13.748203 1543 tablet_loader.cc:96] loaded metadata for tablet d1bc320027bd43b59a7fc77136aeda74 (table default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60])
I20250904 07:53:13.748386 1543 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:13.749100 1543 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:13.749164 1543 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:13.750257 1543 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:13.751039 1543 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d6b92e095cf84479a8896513f2acf0a0: Loaded TSK: 0
I20250904 07:53:13.751995 1543 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:13.752729 1543 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 07:53:13.753402 1543 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 07:53:13.768884 688 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_e71ba3eed2d5429796154c26d9f1c978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.768991 687 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_ad93fa23e5724bf3aaa93e872f55d612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.769210 684 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_dcff30abce7b454b827a08daf85eab8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.769637 686 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_28be2f0ae2d54f5bb0b09107f6069cc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.769711 683 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_3f9daeb5d9b94e808d296ebebcae9e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.770457 681 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_aa64673a1d5e47bdb0b862084daac8ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.770862 685 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_5d14da275d34445abdb109af99d1cc7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.770974 679 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_2fcb873f751947deb8f788e89007069c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.771417 680 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_da1c79fbee874fd58c1a2dfea10b89c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:13.771701 682 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "default.Table_c308554984c24f29b70e853b6e48516b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250904 07:53:13.775456 903 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_5ff476a678d047848753ef2b12175785, message={"server":"","servicePrincipal":"","db":"default","table":"table_5ff476a678d047848753ef2b12175785","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5ff476a678d047848753ef2b12175785\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_5ff476a678d047848753ef2b12175785\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5ff476a678d047848753ef2b12175785\",\"kudu.table_id\":\"783aa6a2ef4a497193e19a03874a4b60\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_aa64673a1d5e47bdb0b862084daac8ca, message={"server":"","servicePrincipal":"","db":"default","table":"table_aa64673a1d5e47bdb0b862084daac8ca","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_aa64673a1d5e47bdb0b862084daac8ca\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_aa64673a1d5e47bdb0b862084daac8ca\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_aa64673a1d5e47bdb0b862084daac8ca\",\"kudu.table_id\":\"02e00655160b41698e3454c7104042bc\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f16f5beadcd google::LogMessage::Fail() at ??:0
@ 0x7f16f5beeb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f16f5bea7cc google::LogMessage::Flush() at ??:0
@ 0x7f16f5beaa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f170c02d7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 07:53:13.790390 1469 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:57621; Asking this server to re-register.
I20250904 07:53:13.791253 1204 heartbeater.cc:461] Registering TS with master...
@ 0x7f170c02c061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250904 07:53:13.793159 1469 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
@ 0x7f170c033cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f170c033b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f16f89532b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f16fbd776db start_thread at ??:0
@ 0x7f16f4a9b71f clone at ??:0
I20250904 07:53:13.836144 1469 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:40181; Asking this server to re-register.
I20250904 07:53:13.836954 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:13.838337 1469 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:53:13.968008 952 connection.cc:537] client connection to 127.0.104.62:34211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:13.968185 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:34211)
W20250904 07:53:13.968472 1346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:34211)
W20250904 07:53:13.969218 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:34211)
I20250904 07:53:14.135833 1558 consensus_queue.cc:1035] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:14.144914 1561 consensus_queue.cc:1035] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:14.353736 1561 consensus_queue.cc:1035] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:14.360988 1558 consensus_queue.cc:1035] T 65c4e146fd534eb2bccc6867be1e755d P 2fc1b8ca505440228c8e9a544e3c549a [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:14.381661 1574 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:14.381882 1574 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:14.383662 1574 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:14.384766 1454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:14.386075 1454 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:14.386265 1454 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:14.386346 1454 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:14.386636 1574 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:15.075197 1581 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:15.075361 1581 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:15.076001 1581 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:15.078122 1454 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:15.078320 1454 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:15.078426 1454 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:15.078675 1581 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:15.608316 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXge7Va/build/asan/bin/kudu pid 1432
I20250904 07:53:15.616490 1499 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 07:53:15.723793 1593 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:15.724083 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXge7Va/build/asan/bin/kudu with pid 1432
I20250904 07:53:16.438380 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=45475
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 07:53:16.752949 1606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:16.753301 1606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:16.753443 1606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:16.761592 1606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:16.761699 1606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:16.761787 1606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:16.761873 1606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:16.772279 1606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:16.773929 1606 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:16.775709 1606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:16.784222 1612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:16.784199 1611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:16.784965 1614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:16.785388 1606 server_base.cc:1047] running on GCE node
I20250904 07:53:16.786170 1606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:16.787568 1606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:16.788836 1606 hybrid_clock.cc:648] HybridClock initialized: now 1756972396788784 us; error 37 us; skew 500 ppm
I20250904 07:53:16.789227 1606 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:16.791477 1606 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:16.792224 1606 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:16.792315 1606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:16.797655 1606 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20250904 07:53:16.800606 1620 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:16.802053 1606 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:16.802211 1606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:16.802937 1606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:16.819170 1606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:16.819994 1606 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:16.820256 1606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:16.839679 1606 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:16.839716 1671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:16.841331 1606 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:16.846179 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 1606
I20250904 07:53:16.848062 1672 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:16.851688 1672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:16.861759 1672 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:16.872407 1672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:16.873294 1672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap complete.
I20250904 07:53:16.875135 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:16.879940 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:16.881172 1672 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:16.881829 1672 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:16.882442 1672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:16.883229 1680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:16.883486 1680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:16.884194 1672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:16.894889 1694 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:16.895080 1694 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: success
I20250904 07:53:16.899137 1694 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: acquiring CA information for follower catalog manager: success
I20250904 07:53:16.901746 1694 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:16.903194 1606 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:16.909093 1606 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:17.272073 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:17.274214 1637 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:48565; Asking this server to re-register.
I20250904 07:53:17.274791 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:17.276212 1637 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:17.378803 1637 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:33131; Asking this server to re-register.
I20250904 07:53:17.379392 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:17.380353 1637 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:17.382278 1637 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:35265; Asking this server to re-register.
I20250904 07:53:17.382802 1204 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:17.383821 1637 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:17.482120 1701 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:17.482400 1701 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:17.484148 1701 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:17.485080 1622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:17.486464 1622 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:17.486701 1622 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:17.486804 1622 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:17.487164 1701 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:18.093017 1706 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:18.093195 1706 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:18.093758 1706 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:18.097275 1622 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:18.097504 1622 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:18.097637 1622 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:18.097926 1706 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:18.932494 1720 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:18.932685 1720 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:18.933285 1720 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:18.935519 1622 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:18.935704 1622 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:18.935797 1622 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:18.936048 1720 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:18.995451 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXge7Va/build/asan/bin/kudu pid 1606
I20250904 07:53:19.002147 1667 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 07:53:19.104822 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXge7Va/build/asan/bin/kudu with pid 1606
I20250904 07:53:20.083010 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=45475
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 07:53:20.401240 1735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:20.401557 1735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:20.401675 1735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:20.409974 1735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:20.410107 1735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:20.410194 1735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:20.410255 1735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:20.419786 1735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:20.421391 1735 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:20.423202 1735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:20.430970 1741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:20.431036 1742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:20.432080 1744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:20.432304 1735 server_base.cc:1047] running on GCE node
I20250904 07:53:20.433045 1735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:20.434477 1735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:20.435688 1735 hybrid_clock.cc:648] HybridClock initialized: now 1756972400435622 us; error 62 us; skew 500 ppm
I20250904 07:53:20.436090 1735 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:20.438359 1735 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:20.439087 1735 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:20.439244 1735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:20.444231 1735 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250904 07:53:20.447036 1750 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:20.448431 1735 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:20.448582 1735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:20.449237 1735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:20.498907 1735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:20.499732 1735 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:20.500012 1735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:20.518906 1735 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:20.518934 1801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:20.520466 1735 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:20.521510 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 1735
I20250904 07:53:20.528065 1802 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:20.532143 1802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:20.539458 1802 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:20.547587 1802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:20.548096 1802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap complete.
I20250904 07:53:20.553050 1802 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:20.553648 1802 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:20.554293 1802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:20.555016 1806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:20.555265 1806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:20.555959 1802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:20.569729 1820 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:20.569857 1820 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: success
I20250904 07:53:20.567451 1735 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:20.573199 1735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:20.573818 1820 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: acquiring CA information for follower catalog manager: success
I20250904 07:53:20.575886 1820 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:20.792431 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:20.794955 1767 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:38535; Asking this server to re-register.
I20250904 07:53:20.795723 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:20.797396 1767 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:20.896263 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:20.897776 1767 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:41609; Asking this server to re-register.
I20250904 07:53:20.898624 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:20.900012 1767 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:20.900207 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:20.901243 1767 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:47989; Asking this server to re-register.
I20250904 07:53:20.901825 1204 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:20.903014 1767 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:20.969029 1806 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:20.969300 1806 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:20.971292 1806 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:20.971899 1752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:20.973646 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:20.973865 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:20.973970 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:20.974493 1806 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:21.538427 1831 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:21.538602 1831 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:21.539253 1831 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:21.541203 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:21.541374 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:21.541497 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:21.541720 1831 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:22.590214 1844 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:22.590390 1844 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:22.590955 1844 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:22.598677 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:22.598933 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:22.599030 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:22.599294 1844 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:23.508605 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40359
--webserver_interface=127.0.104.60
--webserver_port=43075
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 07:53:23.816362 1857 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:23.816552 1857 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:23.818423 1857 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:23.827409 1855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:23.827715 1855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:23.827911 1855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:23.828962 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:23.829166 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:23.829270 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:23.829504 1857 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250904 07:53:23.836553 1855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:23.836681 1855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:23.836792 1855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:23.836860 1855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:23.847211 1855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=43075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:23.848970 1855 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:23.850939 1855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:23.859577 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:23.859490 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:23.859524 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:23.860605 1855 server_base.cc:1047] running on GCE node
I20250904 07:53:23.861322 1855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:23.862743 1855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:23.863955 1855 hybrid_clock.cc:648] HybridClock initialized: now 1756972403863903 us; error 46 us; skew 500 ppm
I20250904 07:53:23.864266 1855 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:23.866382 1855 webserver.cc:480] Webserver started at http://127.0.104.60:43075/ using document root <none> and password file <none>
I20250904 07:53:23.867062 1855 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:23.867179 1855 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:23.872148 1855 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250904 07:53:23.874650 1874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:23.875902 1855 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:23.876058 1855 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
uuid: "1c32cb6c255941f99f1fd72793331fe5"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:23.876686 1855 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:23.888219 1855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:23.889281 1855 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:23.889647 1855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:23.909610 1855 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:40359
I20250904 07:53:23.909626 1927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:40359 every 8 connection(s)
I20250904 07:53:23.911258 1855 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
I20250904 07:53:23.917078 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 1855
I20250904 07:53:23.918192 1928 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:23.922508 1928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Bootstrap starting.
I20250904 07:53:23.934988 1928 log.cc:826] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:23.946336 1928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=7 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:23.946832 1928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Bootstrap complete.
I20250904 07:53:23.951606 1928 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:23.952313 1928 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32cb6c255941f99f1fd72793331fe5, State: Initialized, Role: FOLLOWER
I20250904 07:53:23.952925 1928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:23.953727 1932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:23.954078 1932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:23.954746 1928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:23.977458 1946 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:23.977584 1946 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: loading cluster ID for follower catalog manager: success
I20250904 07:53:23.981925 1946 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: acquiring CA information for follower catalog manager: success
I20250904 07:53:23.983932 1946 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:23.974027 1855 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:23.985606 1855 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:24.297206 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:24.297432 1064 heartbeater.cc:499] Master 127.0.104.60:40359 was elected leader, sending a full tablet report...
I20250904 07:53:24.300896 1893 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:42585; Asking this server to re-register.
I20250904 07:53:24.301709 1064 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:24.303512 1893 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:24.397907 1338 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:24.398161 1338 heartbeater.cc:499] Master 127.0.104.60:40359 was elected leader, sending a full tablet report...
I20250904 07:53:24.398838 1202 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:24.399042 1202 heartbeater.cc:499] Master 127.0.104.60:40359 was elected leader, sending a full tablet report...
I20250904 07:53:24.399390 1893 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:37341; Asking this server to re-register.
I20250904 07:53:24.399932 1338 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:24.400223 1893 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:45983; Asking this server to re-register.
I20250904 07:53:24.400810 1202 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:24.401129 1893 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:24.401885 1893 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:24.407171 1932 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:24.407404 1932 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:24.409144 1932 leader_election.cc:290] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659)
W20250904 07:53:24.409724 1876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:24.411697 1876 leader_election.cc:336] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
I20250904 07:53:24.416589 1777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c32cb6c255941f99f1fd72793331fe5" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" is_pre_election: true
I20250904 07:53:24.417059 1777 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c32cb6c255941f99f1fd72793331fe5 in current term 3: Already voted for candidate d6b92e095cf84479a8896513f2acf0a0 in this term.
I20250904 07:53:24.417846 1875 leader_election.cc:304] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c32cb6c255941f99f1fd72793331fe5; no voters: 3d34d58088e14ab9ba0e5c9dcb0820e1, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:24.418258 1932 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:24.421809 1932 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250904 07:53:25.067485 1958 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:25.067641 1958 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:25.068313 1958 leader_election.cc:290] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659)
I20250904 07:53:25.068877 1777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c32cb6c255941f99f1fd72793331fe5" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" is_pre_election: true
I20250904 07:53:25.069118 1777 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c32cb6c255941f99f1fd72793331fe5 in term 3.
I20250904 07:53:25.069514 1875 leader_election.cc:304] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c32cb6c255941f99f1fd72793331fe5, 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters:
I20250904 07:53:25.069756 1958 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 07:53:25.069865 1958 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:25.069959 1958 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 3 FOLLOWER]: Advancing to term 4
I20250904 07:53:25.071880 1958 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:25.072472 1958 leader_election.cc:290] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 4 election: Requested vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659)
I20250904 07:53:25.073016 1777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c32cb6c255941f99f1fd72793331fe5" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
I20250904 07:53:25.073254 1777 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 3 FOLLOWER]: Advancing to term 4
W20250904 07:53:25.074739 1876 leader_election.cc:336] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:25.075680 1876 leader_election.cc:336] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
I20250904 07:53:25.077550 1777 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c32cb6c255941f99f1fd72793331fe5 in term 4.
I20250904 07:53:25.077970 1875 leader_election.cc:304] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c32cb6c255941f99f1fd72793331fe5, 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:25.078282 1958 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Leader election won for term 4
I20250904 07:53:25.078619 1958 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 LEADER]: Becoming Leader. State: Replica: 1c32cb6c255941f99f1fd72793331fe5, State: Running, Role: LEADER
I20250904 07:53:25.079042 1958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:25.081565 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c32cb6c255941f99f1fd72793331fe5. Latest consensus state: current_term: 4 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:25.081821 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:25.082389 1963 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:25.084668 1963 catalog_manager.cc:671] Loaded metadata for table default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9]
I20250904 07:53:25.085093 1963 catalog_manager.cc:671] Loaded metadata for table default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60]
I20250904 07:53:25.086558 1963 tablet_loader.cc:96] loaded metadata for tablet 65c4e146fd534eb2bccc6867be1e755d (table default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9])
I20250904 07:53:25.086866 1963 tablet_loader.cc:96] loaded metadata for tablet d1bc320027bd43b59a7fc77136aeda74 (table default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60])
I20250904 07:53:25.087064 1963 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:25.087769 1963 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:25.087846 1963 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:25.088898 1963 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:25.089610 1963 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Loaded TSK: 0
I20250904 07:53:25.090461 1963 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:25.091166 1963 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 07:53:25.091868 1963 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 07:53:25.104228 1892 master_service.cc:948] ReplaceTablet: received request to replace tablet 65c4e146fd534eb2bccc6867be1e755d from {username='slave'} at 127.0.0.1:59234
I20250904 07:53:25.104635 1893 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_3f9daeb5d9b94e808d296ebebcae9e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.104954 1890 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_5d14da275d34445abdb109af99d1cc7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.105432 1884 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_da1c79fbee874fd58c1a2dfea10b89c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.106074 1886 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_c308554984c24f29b70e853b6e48516b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.106148 1888 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_e71ba3eed2d5429796154c26d9f1c978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.106458 1891 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_dcff30abce7b454b827a08daf85eab8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.107012 1885 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_aa64673a1d5e47bdb0b862084daac8ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.107733 1887 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_ad93fa23e5724bf3aaa93e872f55d612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:25.108361 1889 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_28be2f0ae2d54f5bb0b09107f6069cc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250904 07:53:25.111409 1945 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_5ff476a678d047848753ef2b12175785, message={"server":"","servicePrincipal":"","db":"default","table":"table_5ff476a678d047848753ef2b12175785","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5ff476a678d047848753ef2b12175785\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_5ff476a678d047848753ef2b12175785\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5ff476a678d047848753ef2b12175785\",\"kudu.table_id\":\"783aa6a2ef4a497193e19a03874a4b60\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_aa64673a1d5e47bdb0b862084daac8ca, message={"server":"","servicePrincipal":"","db":"default","table":"table_aa64673a1d5e47bdb0b862084daac8ca","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_aa64673a1d5e47bdb0b862084daac8ca\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_aa64673a1d5e47bdb0b862084daac8ca\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_aa64673a1d5e47bdb0b862084daac8ca\",\"kudu.table_id\":\"02e00655160b41698e3454c7104042bc\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
W20250904 07:53:25.113461 1876 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 -> Peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Couldn't send request to peer d6b92e095cf84479a8896513f2acf0a0. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f23231a3dcd google::LogMessage::Fail() at ??:0
I20250904 07:53:25.115378 1777 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Refusing update from remote peer 1c32cb6c255941f99f1fd72793331fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
@ 0x7f23231a7b93 google::LogMessage::SendToLog() at ??:0
I20250904 07:53:25.116187 1958 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f23231a37cc google::LogMessage::Flush() at ??:0
@ 0x7f23231a3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f23395e67b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 07:53:25.130882 1966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c32cb6c255941f99f1fd72793331fe5. Latest consensus state: current_term: 4 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
@ 0x7f23395e5061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250904 07:53:25.131181 1966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:25.131506 1970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:25.131844 1970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:25.131788 1964 mvcc.cc:204] Tried to move back new op lower bound from 7196558971331698688 to 7196558971206856704. Current Snapshot: MvccSnapshot[applied={T|T < 7196558971331698688}]
I20250904 07:53:25.133152 1970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:25.133347 1970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f23395eccdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250904 07:53:25.136312 1958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:25.136565 1958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:25.137734 1892 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Sending DeleteTablet for 3 replicas of tablet 65c4e146fd534eb2bccc6867be1e755d
I20250904 07:53:25.139832 1892 catalog_manager.cc:6652] ReplaceTablet: tablet 65c4e146fd534eb2bccc6867be1e755d deleted and replaced by tablet bfdab7963a1b4ecf8f04a323734688df
I20250904 07:53:25.140379 1892 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59234:
name: "default.Table_2fcb873f751947deb8f788e89007069c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f23395ecb59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f2325f0c2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f23293306db start_thread at ??:0
@ 0x7f232205471f clone at ??:0
I20250904 07:53:25.154794 1968 mvcc.cc:204] Tried to move back new op lower bound from 7196558971331698688 to 7196558918762958848. Current Snapshot: MvccSnapshot[applied={T|T < 7196558971331698688}]
W20250904 07:53:25.312927 1979 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:25.145664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.104.3:44879)
0904 07:53:25.154218 (+ 8554us) server_negotiation.cc:207] Beginning negotiation
0904 07:53:25.154224 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0904 07:53:25.154244 (+ 20us) server_negotiation.cc:408] Connection header received
0904 07:53:25.312505 (+158261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57394: BlockingRecv error: recv got EOF from 127.0.0.1:57394 (error 108)
Metrics: {"server-negotiator.queue_time_us":8427,"thread_start_us":9506,"threads_started":1}
W20250904 07:53:25.313148 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.60:40359)
W20250904 07:53:25.313289 952 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:25.313373 1346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.60:40359)
W20250904 07:53:25.314231 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.60:40359)
W20250904 07:53:25.314424 1977 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:25.143771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.0.104.2:44461)
0904 07:53:25.154603 (+ 10832us) server_negotiation.cc:207] Beginning negotiation
0904 07:53:25.154610 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0904 07:53:25.154631 (+ 21us) server_negotiation.cc:408] Connection header received
0904 07:53:25.154706 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0904 07:53:25.154712 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0904 07:53:25.154792 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0904 07:53:25.154902 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0904 07:53:25.312672 (+157770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35664: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":10715,"thread_start_us":11302,"threads_started":1}
I20250904 07:53:25.575006 1735 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:25.575210 1735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:25.634055 1993 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:25.634238 1993 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:25.634850 1993 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:25.636844 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:25.637042 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:25.637131 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:25.637370 1993 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250904 07:53:26.195200 1997 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:26.195390 1997 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:26.195987 1997 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:26.197021 1752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250904 07:53:26.198961 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:26.199170 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:26.199292 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:26.199550 1997 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250904 07:53:26.754139 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40359
--webserver_interface=127.0.104.60
--webserver_port=43075
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 07:53:26.927258 2003 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:26.927440 2003 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:26.928092 2003 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:26.930703 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:26.930953 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:26.931073 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:26.931329 2003 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250904 07:53:27.077224 2002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:27.077526 2002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:27.077611 2002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:27.086067 2002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:27.086191 2002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:27.086253 2002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:27.086318 2002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:27.096558 2002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=43075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:27.098237 2002 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:27.099943 2002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:27.107993 2014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:27.107945 2012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:27.107997 2011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:27.109067 2002 server_base.cc:1047] running on GCE node
I20250904 07:53:27.109826 2002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:27.111289 2002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:27.112495 2002 hybrid_clock.cc:648] HybridClock initialized: now 1756972407112433 us; error 53 us; skew 500 ppm
I20250904 07:53:27.112887 2002 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:27.114864 2002 webserver.cc:480] Webserver started at http://127.0.104.60:43075/ using document root <none> and password file <none>
I20250904 07:53:27.115530 2002 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:27.115619 2002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:27.120605 2002 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20250904 07:53:27.123291 2020 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:27.124953 2002 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:27.125106 2002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
uuid: "1c32cb6c255941f99f1fd72793331fe5"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:27.125739 2002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:27.143359 2002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:27.144140 2002 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:27.144389 2002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:27.163439 2002 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:40359
I20250904 07:53:27.163457 2071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:40359 every 8 connection(s)
I20250904 07:53:27.165041 2002 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-2/data/info.pb
I20250904 07:53:27.171358 2072 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:27.171931 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 2002
I20250904 07:53:27.175173 2072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Bootstrap starting.
I20250904 07:53:27.186228 2072 log.cc:826] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:27.200704 2072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=11 ignored=0} inserts{seen=8 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250904 07:53:27.201243 2072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Bootstrap complete.
I20250904 07:53:27.206104 2072 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:27.208515 2072 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c32cb6c255941f99f1fd72793331fe5, State: Initialized, Role: FOLLOWER
I20250904 07:53:27.209165 2072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:27.209846 2078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:27.210139 2078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:27.210868 2072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:27.221551 2092 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:27.221710 2092 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: loading cluster ID for follower catalog manager: success
I20250904 07:53:27.219744 2002 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:27.225487 2002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:27.226117 2092 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: acquiring CA information for follower catalog manager: success
I20250904 07:53:27.228144 2092 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:27.324733 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:27.327109 2036 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:39295; Asking this server to re-register.
I20250904 07:53:27.327778 1064 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:27.329408 2036 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:27.415951 1202 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:27.416448 1338 heartbeater.cc:344] Connected to a master server at 127.0.104.60:40359
I20250904 07:53:27.417542 2036 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:50187; Asking this server to re-register.
I20250904 07:53:27.417675 2037 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:60139; Asking this server to re-register.
I20250904 07:53:27.418254 1202 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:27.418298 1338 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:27.419601 2037 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:27.419692 2036 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:27.611910 2078 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:27.612161 2078 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:27.614018 2078 leader_election.cc:290] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659)
W20250904 07:53:27.614740 2022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:27.616336 2022 leader_election.cc:336] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
I20250904 07:53:27.620272 1777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c32cb6c255941f99f1fd72793331fe5" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" is_pre_election: true
I20250904 07:53:27.620491 1777 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c32cb6c255941f99f1fd72793331fe5 in term 4.
I20250904 07:53:27.621068 2021 leader_election.cc:304] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c32cb6c255941f99f1fd72793331fe5, 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:27.621421 2078 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250904 07:53:27.621546 2078 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:27.621644 2078 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 4 FOLLOWER]: Advancing to term 5
I20250904 07:53:27.624912 2078 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:27.625551 2078 leader_election.cc:290] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 5 election: Requested vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 3d34d58088e14ab9ba0e5c9dcb0820e1 (127.0.104.61:39659)
I20250904 07:53:27.626097 1777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c32cb6c255941f99f1fd72793331fe5" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
I20250904 07:53:27.626260 1777 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 4 FOLLOWER]: Advancing to term 5
W20250904 07:53:27.626617 2022 leader_election.cc:336] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
I20250904 07:53:27.628014 1777 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c32cb6c255941f99f1fd72793331fe5 in term 5.
I20250904 07:53:27.628407 2021 leader_election.cc:304] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c32cb6c255941f99f1fd72793331fe5, 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:27.628667 2078 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 5 FOLLOWER]: Leader election won for term 5
I20250904 07:53:27.629005 2078 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [term 5 LEADER]: Becoming Leader. State: Replica: 1c32cb6c255941f99f1fd72793331fe5, State: Running, Role: LEADER
I20250904 07:53:27.629453 2078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:27.632040 2099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c32cb6c255941f99f1fd72793331fe5. Latest consensus state: current_term: 5 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:27.632303 2099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:28.104612 1777 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Refusing update from remote peer 1c32cb6c255941f99f1fd72793331fe5: Log matching property violated. Preceding OpId in replica: term: 4 index: 11. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
I20250904 07:53:28.105420 2099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20250904 07:53:28.108790 2102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c32cb6c255941f99f1fd72793331fe5. Latest consensus state: current_term: 5 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:28.109105 2102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:28.111873 2102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:28.112056 2102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:28.111909 2078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "1c32cb6c255941f99f1fd72793331fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:28.112182 2078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 [sys.catalog]: This master's current role is: LEADER
W20250904 07:53:28.112443 2022 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5 -> Peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Couldn't send request to peer d6b92e095cf84479a8896513f2acf0a0. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:28.148506 2101 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:28.150812 2101 catalog_manager.cc:671] Loaded metadata for table default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9]
I20250904 07:53:28.151269 2101 catalog_manager.cc:671] Loaded metadata for table default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60]
I20250904 07:53:28.152925 2101 tablet_loader.cc:96] loaded metadata for tablet bfdab7963a1b4ecf8f04a323734688df (table default.table_406165ecf1be41718f1b7c6d718a1d55 [id=71f73db7e1dc4d9398193a557ca8d3a9])
I20250904 07:53:28.153267 2101 tablet_loader.cc:96] loaded metadata for tablet d1bc320027bd43b59a7fc77136aeda74 (table default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60])
I20250904 07:53:28.153474 2101 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:28.154318 2101 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:28.154390 2101 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:28.155375 2101 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:28.156124 2101 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Loaded TSK: 0
I20250904 07:53:28.156960 2101 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:28.157641 2101 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 07:53:28.158418 2101 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 07:53:28.222184 2037 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_e71ba3eed2d5429796154c26d9f1c978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:28.222560 2029 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_aa64673a1d5e47bdb0b862084daac8ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:28.222714 2033 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_dcff30abce7b454b827a08daf85eab8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:28.221948 2035 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_28be2f0ae2d54f5bb0b09107f6069cc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:28.221968 2034 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_5d14da275d34445abdb109af99d1cc7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:28.223505 2036 master_service.cc:948] ReplaceTablet: received request to replace tablet d1bc320027bd43b59a7fc77136aeda74 from {username='slave'} at 127.0.0.1:36460
I20250904 07:53:28.223443 2032 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_3f9daeb5d9b94e808d296ebebcae9e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:28.223956 2028 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_da1c79fbee874fd58c1a2dfea10b89c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250904 07:53:28.225255 2091 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_5ff476a678d047848753ef2b12175785, message={"server":"","servicePrincipal":"","db":"default","table":"table_5ff476a678d047848753ef2b12175785","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_5ff476a678d047848753ef2b12175785\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_5ff476a678d047848753ef2b12175785\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_5ff476a678d047848753ef2b12175785\",\"kudu.table_id\":\"783aa6a2ef4a497193e19a03874a4b60\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756972392, eventType=CREATE_TABLE, dbName=default, tableName=table_aa64673a1d5e47bdb0b862084daac8ca, message={"server":"","servicePrincipal":"","db":"default","table":"table_aa64673a1d5e47bdb0b862084daac8ca","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_aa64673a1d5e47bdb0b862084daac8ca\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756972392},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/warehouse/table_aa64673a1d5e47bdb0b862084daac8ca\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_aa64673a1d5e47bdb0b862084daac8ca\",\"kudu.table_id\":\"02e00655160b41698e3454c7104042bc\",\"transient_lastDdlTime\":\"1756972392\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e16dcdfd803c4301a0ff0c2aded2b662\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756972392,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250904 07:53:28.224177 2031 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_ad93fa23e5724bf3aaa93e872f55d612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:28.224826 2030 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_c308554984c24f29b70e853b6e48516b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f76777a2dcd google::LogMessage::Fail() at ??:0
@ 0x7f76777a6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f76777a27cc google::LogMessage::Flush() at ??:0
@ 0x7f76777a2a11 google::LogMessage::~LogMessage() at ??:0
I20250904 07:53:28.234416 2036 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1c32cb6c255941f99f1fd72793331fe5: Sending DeleteTablet for 3 replicas of tablet d1bc320027bd43b59a7fc77136aeda74
@ 0x7f768dbe57b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250904 07:53:28.249655 1273 tablet_service.cc:1515] Processing DeleteTablet for tablet d1bc320027bd43b59a7fc77136aeda74 with delete_type TABLET_DATA_DELETED (replaced by tablet d9c7fcc6764440609a0a5d4e6bf726bf) from {username='slave'} at 127.0.0.1:36134
@ 0x7f768dbe4061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250904 07:53:28.254699 2036 catalog_manager.cc:6652] ReplaceTablet: tablet d1bc320027bd43b59a7fc77136aeda74 deleted and replaced by tablet d9c7fcc6764440609a0a5d4e6bf726bf
@ 0x7f768dbebcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250904 07:53:28.256705 2036 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36460:
name: "default.Table_2fcb873f751947deb8f788e89007069c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f768dbebb59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250904 07:53:28.265152 1272 tablet_service.cc:1468] Processing CreateTablet for tablet d9c7fcc6764440609a0a5d4e6bf726bf (DEFAULT_TABLE table=default.table_5ff476a678d047848753ef2b12175785 [id=783aa6a2ef4a497193e19a03874a4b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:28.265686 1272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c7fcc6764440609a0a5d4e6bf726bf. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f767a50b2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f767d92f6db start_thread at ??:0
@ 0x7f767665371f clone at ??:0
I20250904 07:53:28.275828 2123 tablet_bootstrap.cc:492] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b: Bootstrap starting.
I20250904 07:53:28.276425 2119 tablet_replica.cc:331] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: stopping tablet replica
I20250904 07:53:28.276885 2119 raft_consensus.cc:2241] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:28.277340 2119 raft_consensus.cc:2270] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:28.278275 2123 tablet_bootstrap.cc:654] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:28.278903 2119 ts_tablet_manager.cc:1905] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 07:53:28.286624 2123 tablet_bootstrap.cc:492] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b: No bootstrap required, opened a new log
I20250904 07:53:28.286772 2123 ts_tablet_manager.cc:1397] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20250904 07:53:28.287175 2119 ts_tablet_manager.cc:1918] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250904 07:53:28.287295 2119 log.cc:1199] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: Deleting WAL directory at /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/ts-2/wal/wals/d1bc320027bd43b59a7fc77136aeda74
I20250904 07:53:28.287603 2123 raft_consensus.cc:357] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:53:28.287786 2119 ts_tablet_manager.cc:1939] T d1bc320027bd43b59a7fc77136aeda74 P 7fce49d1a43d4cdbbfde7f0692eee50b: Deleting consensus metadata
I20250904 07:53:28.287811 2123 raft_consensus.cc:383] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:28.287899 2123 raft_consensus.cc:738] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fce49d1a43d4cdbbfde7f0692eee50b, State: Initialized, Role: FOLLOWER
I20250904 07:53:28.288164 2123 consensus_queue.cc:260] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:53:28.290241 2123 ts_tablet_manager.cc:1428] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
W20250904 07:53:28.438891 1229 connection.cc:537] server connection from 127.0.0.1:36134 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:28.439101 1091 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:28.439373 2113 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:28.237163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.104.1:41509)
0904 07:53:28.276518 (+ 39355us) server_negotiation.cc:207] Beginning negotiation
0904 07:53:28.276527 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0904 07:53:28.276550 (+ 23us) server_negotiation.cc:408] Connection header received
0904 07:53:28.276627 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0904 07:53:28.276636 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0904 07:53:28.276733 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0904 07:53:28.276859 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0904 07:53:28.438988 (+162129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37178: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":39189,"thread_start_us":39121,"threads_started":1}
W20250904 07:53:28.439432 2116 negotiation.cc:337] Failed RPC negotiation. Trace:
0904 07:53:28.241325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.0.104.2:44461)
0904 07:53:28.269353 (+ 28028us) server_negotiation.cc:207] Beginning negotiation
0904 07:53:28.269362 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0904 07:53:28.269386 (+ 24us) server_negotiation.cc:408] Connection header received
0904 07:53:28.269477 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0904 07:53:28.269482 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0904 07:53:28.269581 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0904 07:53:28.269705 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0904 07:53:28.272599 (+ 2894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0904 07:53:28.274266 (+ 1667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0904 07:53:28.439039 (+164773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33530: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":27889,"thread_start_us":28566,"threads_started":1}
W20250904 07:53:28.439620 952 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:28.440394 1227 connection.cc:537] client connection to 127.0.104.60:40359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:28.441018 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.60:40359)
W20250904 07:53:28.441299 1346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.60:40359)
W20250904 07:53:28.442127 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.60:40359)
W20250904 07:53:28.735726 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:28.763232 2133 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:28.763415 2133 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:28.764066 2133 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:28.766130 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:28.766335 1752 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:28.766424 1752 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:28.766767 2133 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:29.108355 2139 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:29.108523 2139 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:53:29.109908 2139 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:53:29.117378 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:53:29.117612 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:53:29.117708 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:53:29.118028 2139 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:53:29.389102 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXge7Va/build/asan/bin/kudu pid 1735
I20250904 07:53:29.395198 1797 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 07:53:29.505491 2146 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c32cb6c255941f99f1fd72793331fe5)
I20250904 07:53:29.505751 2146 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:29.506155 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXge7Va/build/asan/bin/kudu with pid 1735
I20250904 07:53:30.350980 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=45475
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 07:53:30.669209 2155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:30.669533 2155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:30.669690 2155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:30.678671 2155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:30.678774 2155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:30.678881 2155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:30.678953 2155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:30.688395 2155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:30.690048 2155 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:30.691872 2155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:30.700098 2164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:30.700217 2163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:30.700903 2166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:30.701249 2155 server_base.cc:1047] running on GCE node
I20250904 07:53:30.701974 2155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:30.703486 2155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:30.704694 2155 hybrid_clock.cc:648] HybridClock initialized: now 1756972410704628 us; error 52 us; skew 500 ppm
I20250904 07:53:30.705020 2155 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:30.707311 2155 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:30.708038 2155 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:30.708148 2155 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:30.713400 2155 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20250904 07:53:30.716190 2172 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:30.717464 2155 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:30.717613 2155 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:30.718385 2155 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:30.747954 2155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:30.748795 2155 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:30.749037 2155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:30.767442 2224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:30.767448 2155 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:30.769227 2155 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:30.775264 2225 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:30.778196 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 2155
I20250904 07:53:30.778694 2225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:30.788291 2225 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:30.804844 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:30.805121 2225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=9 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:30.805589 2225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap complete.
I20250904 07:53:30.810519 2225 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:30.811192 2225 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:30.811725 2225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 5.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:30.812453 2229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:30.812677 2229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:30.813342 2225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:30.825943 2243 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:30.826131 2243 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: success
I20250904 07:53:30.822474 2155 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:30.828140 2155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:30.830463 2243 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: acquiring CA information for follower catalog manager: success
I20250904 07:53:30.833036 2243 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:31.016040 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:31.018309 2190 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:33071; Asking this server to re-register.
I20250904 07:53:31.018973 1204 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:31.020720 2190 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:31.241959 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:31.243522 2190 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:37625; Asking this server to re-register.
I20250904 07:53:31.244027 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:31.245105 2190 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:31.270860 2229 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:31.271118 2229 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:31.272915 2229 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:31.273655 2174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:31.275288 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:31.275490 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:31.275586 2174 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:31.275920 2229 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:31.307979 2190 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:60033; Asking this server to re-register.
I20250904 07:53:31.308568 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:31.309697 2190 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:31.863539 2255 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:31.863755 2255 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:31.864493 2255 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:31.866639 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:31.866820 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:31.866927 2174 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:31.867169 2255 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:32.541160 2263 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:32.541326 2263 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:32.542035 2263 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:32.547109 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:32.547343 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:32.547438 2174 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:32.547670 2263 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:33.071795 2269 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:33.072043 2269 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:33.072762 2269 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:33.074880 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:33.075079 2174 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:33.075204 2174 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:33.075588 2269 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:33.189957 2273 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:33.190169 2273 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:53:33.190872 2273 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:53:33.191710 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:53:33.191960 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:53:33.192109 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:53:33.192354 2273 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:53:33.238808 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXge7Va/build/asan/bin/kudu pid 2155
I20250904 07:53:33.249599 2220 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 07:53:33.350435 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXge7Va/build/asan/bin/kudu with pid 2155
I20250904 07:53:34.152812 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=45475
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250904 07:53:34.464255 2288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:34.464550 2288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:34.464676 2288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:34.472543 2288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:34.472640 2288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:34.472709 2288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:34.472782 2288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:34.482468 2288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:34.484139 2288 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:34.485916 2288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:34.493850 2294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:34.493844 2293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:34.494621 2296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:34.495079 2288 server_base.cc:1047] running on GCE node
I20250904 07:53:34.495821 2288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:34.497269 2288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:34.498538 2288 hybrid_clock.cc:648] HybridClock initialized: now 1756972414498482 us; error 40 us; skew 500 ppm
I20250904 07:53:34.498936 2288 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:34.501232 2288 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:34.502089 2288 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:34.502208 2288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:34.507663 2288 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20250904 07:53:34.510907 2303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:34.512269 2288 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250904 07:53:34.512430 2288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:34.513185 2288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:34.558199 2288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:34.559073 2288 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:34.559361 2288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:34.578831 2288 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:34.578871 2357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:34.580454 2288 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:34.581223 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 2288
I20250904 07:53:34.588759 2358 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:34.595000 2358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:34.604328 2358 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:34.617674 2358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=9 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:34.618278 2358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap complete.
I20250904 07:53:34.623973 2358 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:34.624720 2358 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:34.625450 2358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 5.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:34.626389 2365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:34.626718 2365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:34.627408 2358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:34.629323 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:34.641716 2379 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:34.641858 2379 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: success
I20250904 07:53:34.637027 2288 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:34.643146 2288 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:34.645707 2379 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: acquiring CA information for follower catalog manager: success
I20250904 07:53:34.648164 2379 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:34.861188 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:34.863718 2323 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:51177; Asking this server to re-register.
I20250904 07:53:34.864470 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:34.866487 2323 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:35.037933 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:35.039166 2323 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:41323; Asking this server to re-register.
I20250904 07:53:35.039719 1204 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:35.040818 2323 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:35.133503 2323 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:45411; Asking this server to re-register.
I20250904 07:53:35.134114 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:35.135263 2323 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:35.238090 2384 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:35.238373 2384 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:35.240211 2384 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:35.241225 2305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:35.242596 2305 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:35.242914 2305 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:35.243045 2305 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:35.243383 2384 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:35.897182 2390 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:35.897364 2390 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:35.898087 2390 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:35.902992 2305 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:35.903218 2305 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:35.903327 2305 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:35.903627 2390 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:36.870174 2401 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:36.870373 2401 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:36.870903 2401 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:36.872992 2305 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:36.873219 2305 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:36.873312 2305 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:36.873564 2401 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:37.084868 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXge7Va/build/asan/bin/kudu pid 2288
I20250904 07:53:37.092085 2353 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 07:53:37.193606 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXge7Va/build/asan/bin/kudu with pid 2288
I20250904 07:53:37.752306 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=45475
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 07:53:37.845185 2415 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:37.845381 2415 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:53:37.846081 2415 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:53:37.847033 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:53:37.847251 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:53:37.847398 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:53:37.847637 2415 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:53:38.076967 2414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:38.077292 2414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:38.077436 2414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:38.085783 2414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:38.085886 2414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:38.085945 2414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:38.086035 2414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:38.096035 2414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:38.097699 2414 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:38.099610 2414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:38.108225 2423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:38.108209 2420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:38.108340 2421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:38.109598 2414 server_base.cc:1047] running on GCE node
I20250904 07:53:38.110898 2414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:38.112681 2414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:38.113924 2414 hybrid_clock.cc:648] HybridClock initialized: now 1756972418113873 us; error 53 us; skew 500 ppm
I20250904 07:53:38.114305 2414 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:38.116531 2414 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:38.117286 2414 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:38.117386 2414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:38.124902 2414 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20250904 07:53:38.127779 2432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:38.129189 2414 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:38.129357 2414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:38.130172 2414 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:38.147404 2414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:38.148231 2414 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:38.148495 2414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:38.166607 2414 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:38.166620 2483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:38.168316 2414 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:38.170639 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 2414
I20250904 07:53:38.175550 2484 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:38.179816 2484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:38.188843 2484 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:38.202204 2484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=9 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:38.202694 2484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap complete.
I20250904 07:53:38.207527 2484 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:38.208151 2484 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:38.208802 2484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 5.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:38.209570 2489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:38.209822 2489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:38.210614 2484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:38.222563 2503 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:38.222765 2503 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: success
I20250904 07:53:38.226708 2503 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: acquiring CA information for follower catalog manager: success
I20250904 07:53:38.229357 2503 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:38.230472 2414 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:38.235639 2414 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:38.385483 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:38.388031 2449 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:40511; Asking this server to re-register.
I20250904 07:53:38.388744 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:38.390468 2449 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
W20250904 07:53:38.426456 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 6: this message will repeat every 5th retry.
I20250904 07:53:38.433432 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:38.435331 2449 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:49395; Asking this server to re-register.
I20250904 07:53:38.435982 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:38.437311 2449 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:38.559131 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:38.560333 2449 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:52857; Asking this server to re-register.
I20250904 07:53:38.560971 1204 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:38.562381 2449 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:38.600553 2489 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:38.600822 2489 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:38.602870 2489 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:38.603579 2434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:38.605191 2434 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:38.605412 2434 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:38.605499 2434 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:38.605849 2489 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:39.250584 2511 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:39.250756 2511 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:39.251482 2511 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:39.255299 2434 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:39.255496 2434 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:39.255599 2434 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:39.255858 2511 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:40.278510 2521 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:40.278738 2521 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:40.279512 2521 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:40.281951 2434 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:40.282200 2434 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:40.282292 2434 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:40.282575 2521 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:40.289183 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskXge7Va/build/asan/bin/kudu pid 2414
I20250904 07:53:40.299727 2479 generic_service.cc:196] Checking for leaks (request via RPC)
I20250904 07:53:40.399801 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXge7Va/build/asan/bin/kudu with pid 2414
I20250904 07:53:41.397375 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXge7Va/build/asan/bin/kudu
/tmp/dist-test-taskXge7Va/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--webserver_interface=127.0.104.61
--webserver_port=45475
--builtin_ntp_servers=127.0.104.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250904 07:53:41.625866 2538 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:41.626102 2538 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:53:41.626796 2538 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:53:41.627719 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:53:41.627923 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:53:41.628101 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:53:41.628388 2538 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:53:41.721127 2537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 07:53:41.721416 2537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 07:53:41.721542 2537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 07:53:41.730054 2537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 07:53:41.730158 2537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 07:53:41.730235 2537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 07:53:41.730311 2537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 07:53:41.739692 2537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39003
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 07:43:24 UTC on 24a791456cd2
build id 7892
ASAN enabled
I20250904 07:53:41.741310 2537 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:53:41.743151 2537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:41.751148 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.751556 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.751149 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:41.752246 2537 server_base.cc:1047] running on GCE node
I20250904 07:53:41.752964 2537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 07:53:41.754374 2537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 07:53:41.755566 2537 hybrid_clock.cc:648] HybridClock initialized: now 1756972421755533 us; error 25 us; skew 500 ppm
I20250904 07:53:41.755908 2537 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:41.757965 2537 webserver.cc:480] Webserver started at http://127.0.104.61:45475/ using document root <none> and password file <none>
I20250904 07:53:41.758795 2537 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:41.758899 2537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:41.763978 2537 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250904 07:53:41.766678 2552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.767994 2537 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:41.768144 2537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1"
format_stamp: "Formatted at 2025-09-04 07:53:09 on dist-test-slave-jtcb"
I20250904 07:53:41.768779 2537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:41.784699 2537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:41.785516 2537 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:53:41.785795 2537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:41.805359 2537 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39659
I20250904 07:53:41.805377 2603 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39659 every 8 connection(s)
I20250904 07:53:41.807014 2537 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXge7Va/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756972374302231-416-0/minicluster-data/master-1/data/info.pb
I20250904 07:53:41.813266 2604 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:41.815438 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXge7Va/build/asan/bin/kudu as pid 2537
I20250904 07:53:41.816591 2604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap starting.
I20250904 07:53:41.826997 2604 log.cc:826] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Log is configured to *not* fsync() on all Append() calls
I20250904 07:53:41.843576 2604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=9 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:41.844137 2604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: Bootstrap complete.
I20250904 07:53:41.848994 2604 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:41.849625 2604 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d34d58088e14ab9ba0e5c9dcb0820e1, State: Initialized, Role: FOLLOWER
I20250904 07:53:41.850279 2604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 5.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:41.851032 2608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } } }
I20250904 07:53:41.851279 2608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 07:53:41.852010 2604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:41.863684 2622 catalog_manager.cc:1261] Loaded cluster ID: e16dcdfd803c4301a0ff0c2aded2b662
I20250904 07:53:41.863884 2622 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: loading cluster ID for follower catalog manager: success
I20250904 07:53:41.868021 2622 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: acquiring CA information for follower catalog manager: success
I20250904 07:53:41.870599 2622 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 07:53:41.871688 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:41.877856 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:41.885324 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:41.887953 2569 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" instance_seqno: 1756972390901859) as {username='slave'} at 127.0.104.1:53505; Asking this server to re-register.
I20250904 07:53:41.888613 1066 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:41.890133 2569 ts_manager.cc:194] Registered new tserver with Master: 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
I20250904 07:53:41.928241 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:41.929597 2569 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" instance_seqno: 1756972391916023) as {username='slave'} at 127.0.104.3:33419; Asking this server to re-register.
I20250904 07:53:41.930171 1340 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:41.931545 2569 ts_manager.cc:194] Registered new tserver with Master: 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879)
I20250904 07:53:42.079973 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.61:39659
I20250904 07:53:42.081460 2569 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "45f0af8a16f14b8b83950e205831688f" instance_seqno: 1756972391435041) as {username='slave'} at 127.0.104.2:47127; Asking this server to re-register.
I20250904 07:53:42.082087 1204 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:42.083377 2569 ts_manager.cc:194] Registered new tserver with Master: 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461)
I20250904 07:53:42.343694 2608 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:42.344012 2608 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:42.345948 2608 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:42.346613 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:42.348141 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:42.348320 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:42.348393 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:42.348696 2608 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:42.961851 2634 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:42.962086 2634 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:42.962780 2634 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:42.968299 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:42.968464 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:42.968567 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:42.968832 2634 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:43.967844 2645 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:43.968061 2645 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:43.968703 2645 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:43.974345 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:43.974575 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:43.974668 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:43.974962 2645 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:44.018769 416 master-stress-test.cc:577] Killing test thread 1/25
I20250904 07:53:45.527354 2661 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:45.527529 2661 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:45.528192 2661 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:45.530829 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:45.531016 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:45.531141 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:45.531414 2661 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:46.315064 2671 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:46.315246 2671 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:53:46.316088 2671 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:53:46.317011 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:53:46.317183 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:53:46.317269 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:53:46.317471 2671 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:53:46.412794 2672 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:46.412977 2672 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:46.413540 2672 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:46.420951 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:46.421213 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:46.421330 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:46.421618 2672 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:46.879037 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:46.879248 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:53:49.329741 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 11: this message will repeat every 5th retry.
I20250904 07:53:50.089426 2695 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:50.089663 2695 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:50.090335 2695 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:50.091562 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250904 07:53:50.095196 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:50.095435 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:50.095532 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:50.095872 2695 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:51.880785 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:51.880999 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:53:54.524705 2725 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:54.524959 2725 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:53:54.525734 2725 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:53:54.528033 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:53:54.528218 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:53:54.528306 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:53:54.528564 2725 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:56.882493 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:53:56.882714 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:53:59.023738 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 16: this message will repeat every 5th retry.
I20250904 07:54:00.845604 2767 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:00.845799 2767 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:54:00.846422 2767 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:54:00.847581 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 07:54:00.851125 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:00.851315 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:54:00.851429 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:54:00.851681 2767 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:54:01.884056 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:01.884248 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:54:03.484467 2789 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:03.484655 2789 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:54:03.485297 2789 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:54:03.491073 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:03.491317 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:54:03.491421 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:54:03.491653 2789 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:54:04.688002 2802 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:04.688201 2802 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:54:04.688905 2802 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:54:04.689673 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:54:04.689806 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:54:04.689927 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:54:04.690199 2802 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:54:06.885469 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:06.885658 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:54:09.445691 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 21: this message will repeat every 5th retry.
I20250904 07:54:11.887120 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:11.887308 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:54:12.507421 1203 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:34211 (120 consecutive failures): Network error: Failed to ping master at 127.0.104.62:34211: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:12.539674 1338 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (89 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:54:12.598186 1065 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:34211 (142 consecutive failures): Network error: Failed to ping master at 127.0.104.62:34211: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
I20250904 07:54:14.293551 2862 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:14.293753 2862 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:54:14.294420 2862 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:54:14.295672 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 07:54:14.299010 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:14.299202 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:54:14.299290 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:54:14.299539 2862 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:54:16.888749 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:16.888948 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:54:19.522552 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 26: this message will repeat every 5th retry.
I20250904 07:54:20.203647 2903 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:20.203845 2903 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:54:20.204738 2903 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:54:20.205637 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:54:20.205814 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:54:20.205909 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:54:20.206163 2903 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:54:21.890430 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:21.890596 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:54:26.892191 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:26.892405 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:54:29.639124 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 31: this message will repeat every 5th retry.
I20250904 07:54:31.829697 2974 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:31.829898 2974 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:54:31.830832 2974 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:54:31.832358 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:54:31.833896 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:31.834223 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:54:31.834335 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:54:31.834647 2974 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:54:31.898942 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:31.899153 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:54:36.900692 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:36.900885 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:54:37.158666 3008 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:37.158878 3008 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:54:37.159734 3008 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:54:37.160609 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:54:37.160759 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:54:37.160837 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:54:37.161104 3008 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:54:37.214721 3011 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:37.214926 3011 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:54:37.215631 3011 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:54:37.216421 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:54:37.217801 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:37.217944 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:54:37.218055 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:54:37.218283 3011 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:54:38.396374 3019 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:38.396565 3019 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:54:38.397192 3019 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:54:38.403159 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:38.403380 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:54:38.403484 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:54:38.403718 3019 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250904 07:54:38.445593 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 36: this message will repeat every 5th retry.
I20250904 07:54:41.902590 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:41.902853 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:54:46.904528 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:46.904711 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:54:47.989295 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 41: this message will repeat every 5th retry.
I20250904 07:54:51.906293 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:51.906484 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:54:54.925040 3124 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:54.925252 3124 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:54:54.926307 3124 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:54:54.927399 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 07:54:54.928737 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:54:54.928911 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:54:54.929046 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:54:54.929297 3124 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:54:55.691547 3131 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:55.691785 3131 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:54:55.692621 3131 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:54:55.693619 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:54:55.693801 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:54:55.693888 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:54:55.694172 3131 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:54:56.907928 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:54:56.908138 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:54:58.523355 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 46: this message will repeat every 5th retry.
I20250904 07:55:00.347707 3159 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:00.348011 3159 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:55:00.348814 3159 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:55:00.349567 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:55:00.349740 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:55:00.349853 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:55:00.350139 3159 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:55:01.909644 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:01.909833 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:06.911438 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:06.911641 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:55:08.283167 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 51: this message will repeat every 5th retry.
I20250904 07:55:09.915099 3221 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:09.915262 3221 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:55:09.915815 3221 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:55:09.916980 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:55:09.919135 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:55:09.919337 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:55:09.919448 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:55:09.919709 3221 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:55:11.913064 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:11.913285 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:55:12.657557 1203 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:34211 (240 consecutive failures): Network error: Failed to ping master at 127.0.104.62:34211: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:55:12.684468 1338 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (209 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:55:12.902308 1064 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (262 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:55:16.914794 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:16.914963 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:17.082453 3270 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:17.082686 3270 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:55:17.083617 3270 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:55:17.084587 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:55:17.084745 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:55:17.084872 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:55:17.085183 3270 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:55:18.800961 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 56: this message will repeat every 5th retry.
I20250904 07:55:20.688968 3291 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:20.689143 3291 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:55:20.689837 3291 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:55:20.690879 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:55:20.691025 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:55:20.691190 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:55:20.691416 3291 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:55:21.916621 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:21.916818 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:26.918637 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:26.918860 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:55:29.053046 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 61: this message will repeat every 5th retry.
I20250904 07:55:29.357029 3344 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:29.357240 3344 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:55:29.358150 3344 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:55:29.359683 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:55:29.361260 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:55:29.361467 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:55:29.361588 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:55:29.361831 3344 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:55:31.920468 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:31.920672 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:36.922648 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:36.922829 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:37.176136 3408 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:37.176338 3408 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:55:37.177094 3408 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:55:37.177910 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:55:37.178079 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:55:37.178181 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:55:37.178364 3408 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:55:39.703292 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 66: this message will repeat every 5th retry.
I20250904 07:55:41.924207 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:41.924392 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:41.955163 2558 maintenance_manager.cc:419] P 3d34d58088e14ab9ba0e5c9dcb0820e1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033362
I20250904 07:55:41.996452 2557 maintenance_manager.cc:643] P 3d34d58088e14ab9ba0e5c9dcb0820e1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.041s user 0.015s sys 0.009s Metrics: {"bytes_written":7831,"cfile_init":1,"dirs.queue_time_us":383,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1005,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":9,"thread_start_us":497,"threads_started":2,"wal-append.queue_time_us":373}
I20250904 07:55:41.997841 2558 maintenance_manager.cc:419] P 3d34d58088e14ab9ba0e5c9dcb0820e1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 627 bytes on disk
I20250904 07:55:41.998673 2557 maintenance_manager.cc:643] P 3d34d58088e14ab9ba0e5c9dcb0820e1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20250904 07:55:46.925690 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:46.925876 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:49.250012 3533 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:49.250182 3533 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:55:49.250885 3533 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:55:49.251941 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:55:49.254501 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:55:49.254683 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:55:49.254786 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:55:49.255043 3533 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250904 07:55:49.379633 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 71: this message will repeat every 5th retry.
I20250904 07:55:51.594640 3556 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:51.594851 3556 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:55:51.595490 3556 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:55:51.596400 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:55:51.596565 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:55:51.596681 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:55:51.596920 3556 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:55:51.927448 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:51.927645 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:55:52.641201 3569 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:55:52.641381 3569 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:55:52.641963 3569 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:55:52.645416 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:55:52.645658 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:55:52.645742 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:55:52.646023 3569 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:55:56.929388 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:55:56.929597 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:55:59.143741 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 76: this message will repeat every 5th retry.
I20250904 07:56:00.151207 3643 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:00.151407 3643 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:56:00.152251 3643 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:56:00.153028 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:56:00.153153 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:56:00.153330 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:56:00.153580 3643 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:56:01.931347 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:01.931586 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:06.148773 3700 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:06.149013 3700 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:56:06.149713 3700 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:56:06.150993 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 07:56:06.153873 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:56:06.154120 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:56:06.154213 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:56:06.154517 3700 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:56:06.932991 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:06.933202 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:08.357597 3726 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:08.357800 3726 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:56:08.358525 3726 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:56:08.359514 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:56:08.359786 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:56:08.359925 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:56:08.360225 3726 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:56:08.713958 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 81: this message will repeat every 5th retry.
I20250904 07:56:11.934774 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:11.934990 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:56:12.801438 1203 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:34211 (360 consecutive failures): Network error: Failed to ping master at 127.0.104.62:34211: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:56:12.824184 1338 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (329 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:56:13.213568 1064 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (413 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:56:16.936524 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:16.936722 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:56:18.309041 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 86: this message will repeat every 5th retry.
I20250904 07:56:21.938530 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:21.938709 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:24.117074 3874 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:24.117321 3874 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:56:24.118235 3874 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:56:24.119837 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:56:24.121138 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:56:24.121344 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:56:24.121459 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:56:24.121733 3874 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:56:25.104632 3887 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:25.104835 3887 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:56:25.105743 3887 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:56:25.106704 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:56:25.106866 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:56:25.106988 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:56:25.107208 3887 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:56:26.940135 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:26.940336 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:56:28.750149 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 91: this message will repeat every 5th retry.
I20250904 07:56:31.941748 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:31.941941 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:36.943668 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:36.943877 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:56:38.946954 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 96: this message will repeat every 5th retry.
I20250904 07:56:41.356997 4033 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:41.357214 4033 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:56:41.358143 4033 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:56:41.359154 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:56:41.359364 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:56:41.359531 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:56:41.359915 4033 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:56:41.945266 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:41.945436 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:43.036859 4046 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:43.037068 4046 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:56:43.037827 4046 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:56:43.039156 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:56:43.043182 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:56:43.043406 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:56:43.043489 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:56:43.043720 4046 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:56:46.948081 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:46.948318 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:47.305970 4088 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:47.306190 4088 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:56:47.306839 4088 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:56:47.308791 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:56:47.308956 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:56:47.309075 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:56:47.309346 4088 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250904 07:56:49.252509 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 101: this message will repeat every 5th retry.
I20250904 07:56:51.949810 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:51.950059 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:56.951577 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:56:56.951761 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:56:57.735677 4179 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:56:57.735904 4179 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:56:57.736832 4179 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:56:57.737722 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:56:57.737890 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:56:57.738051 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:56:57.738283 4179 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:56:59.115079 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 106: this message will repeat every 5th retry.
I20250904 07:57:00.099792 4202 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:00.100003 4202 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:57:00.100718 4202 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:57:00.101966 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 07:57:00.106727 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:57:00.106958 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:57:00.107522 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:57:00.107828 4202 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:57:01.953270 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:01.953490 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:06.954854 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:06.955065 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:57:08.853684 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 111: this message will repeat every 5th retry.
I20250904 07:57:11.956390 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:11.956596 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:12.464263 4314 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:12.464530 4314 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:57:12.465227 4314 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:57:12.466138 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:57:12.466351 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:57:12.466496 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:57:12.466753 4314 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:57:12.946688 1203 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:34211 (480 consecutive failures): Network error: Failed to ping master at 127.0.104.62:34211: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:57:12.968051 1338 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (449 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:57:13.369338 1064 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (562 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:57:16.138607 4347 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:16.138809 4347 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:57:16.139539 4347 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:57:16.140395 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:57:16.140537 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:57:16.140681 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:57:16.140893 4347 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:57:16.957983 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:16.958248 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:18.530309 4368 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:18.530520 4368 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:57:18.531345 4368 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:57:18.532517 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:57:18.534297 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:57:18.534464 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:57:18.534551 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:57:18.534776 4368 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250904 07:57:18.791327 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 116: this message will repeat every 5th retry.
I20250904 07:57:19.366266 4380 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:19.366468 4380 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:57:19.367157 4380 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:57:19.367976 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:57:19.368167 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:57:19.368296 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:57:19.368518 4380 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:57:21.959736 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:21.959915 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:26.961383 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:26.961591 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:28.363272 4460 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:28.363473 4460 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:57:28.364074 4460 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:57:28.365046 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:57:28.365192 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:57:28.365335 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:57:28.365589 4460 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:57:28.815277 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 121: this message will repeat every 5th retry.
I20250904 07:57:31.963092 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:31.963325 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:32.843082 4495 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:32.843320 4495 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:57:32.844002 4495 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:57:32.845010 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:57:32.846741 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:57:32.846896 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:57:32.846988 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:57:32.847220 4495 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:57:36.168887 4525 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:36.169142 4525 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:57:36.169809 4525 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:57:36.172427 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:57:36.172621 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:57:36.172706 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:57:36.172979 4525 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:57:36.964861 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:36.965109 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:57:38.953125 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 126: this message will repeat every 5th retry.
I20250904 07:57:41.966662 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:41.966871 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:46.968220 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:46.968403 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:46.990396 4624 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:46.990617 4624 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:57:46.991569 4624 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:57:46.992622 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:57:46.992856 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:57:46.992997 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:57:46.993242 4624 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250904 07:57:49.295001 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 131: this message will repeat every 5th retry.
I20250904 07:57:51.969833 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:51.970081 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:57:53.326750 4679 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:57:53.326954 4679 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:57:53.327751 4679 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:57:53.329064 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 07:57:53.331669 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:57:53.331898 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:57:53.331983 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:57:53.332302 4679 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:57:56.971457 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:57:56.971657 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:57:59.111788 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 136: this message will repeat every 5th retry.
I20250904 07:58:01.973165 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:58:01.973340 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 07:58:03.314759 4769 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:58:03.314980 4769 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:58:03.315713 4769 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:58:03.316691 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250904 07:58:03.318150 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:58:03.318409 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:58:03.318521 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:58:03.318801 4769 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:58:05.523211 4795 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:58:05.523402 4795 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:58:05.524242 4795 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:58:05.525229 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:58:05.525388 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:58:05.525511 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:58:05.525733 4795 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:58:06.974673 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:58:06.974871 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:58:07.860553 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 141: this message will repeat every 5th retry.
I20250904 07:58:07.896351 4815 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:58:07.896561 4815 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:58:07.897192 4815 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:58:07.899369 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:58:07.899525 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:58:07.899605 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:58:07.899828 4815 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:58:11.976327 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:58:11.976547 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:58:13.087153 1203 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:34211 (600 consecutive failures): Network error: Failed to ping master at 127.0.104.62:34211: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:58:13.105841 1338 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40359 (569 consecutive failures): Network error: Failed to ping master at 127.0.104.60:40359: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:58:13.665450 1065 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:34211 (745 consecutive failures): Network error: Failed to ping master at 127.0.104.62:34211: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
I20250904 07:58:16.623227 4888 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:58:16.623442 4888 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:58:16.624091 4888 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:58:16.624984 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:58:16.625149 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:58:16.625262 1227 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:58:16.625485 4888 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:58:16.980834 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:58:16.981002 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:58:18.257059 951 consensus_peers.cc:489] T d1bc320027bd43b59a7fc77136aeda74 P 2fc1b8ca505440228c8e9a544e3c549a -> Peer 7fce49d1a43d4cdbbfde7f0692eee50b (127.0.104.3:44879): Couldn't send request to peer 7fce49d1a43d4cdbbfde7f0692eee50b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bc320027bd43b59a7fc77136aeda74. This is attempt 146: this message will repeat every 5th retry.
I20250904 07:58:18.359508 4904 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:58:18.359735 4904 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b92e095cf84479a8896513f2acf0a0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34211 } } peers { permanent_uuid: "3d34d58088e14ab9ba0e5c9dcb0820e1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39659 } } peers { permanent_uuid: "1c32cb6c255941f99f1fd72793331fe5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40359 } }
I20250904 07:58:18.360404 4904 leader_election.cc:290] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211), 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359)
W20250904 07:58:18.361548 2554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250904 07:58:18.362896 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d6b92e095cf84479a8896513f2acf0a0 (127.0.104.62:34211): Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111)
W20250904 07:58:18.363549 2554 leader_election.cc:336] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c32cb6c255941f99f1fd72793331fe5 (127.0.104.60:40359): Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
I20250904 07:58:18.363672 2554 leader_election.cc:304] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d34d58088e14ab9ba0e5c9dcb0820e1; no voters: 1c32cb6c255941f99f1fd72793331fe5, d6b92e095cf84479a8896513f2acf0a0
I20250904 07:58:18.363907 4904 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3d34d58088e14ab9ba0e5c9dcb0820e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:58:19.686317 4917 raft_consensus.cc:491] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:58:19.686542 4917 raft_consensus.cc:513] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f0af8a16f14b8b83950e205831688f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44461 } } peers { permanent_uuid: "7fce49d1a43d4cdbbfde7f0692eee50b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44879 } } peers { permanent_uuid: "2fc1b8ca505440228c8e9a544e3c549a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41509 } }
I20250904 07:58:19.687222 4917 leader_election.cc:290] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461), 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509)
W20250904 07:58:19.688081 1227 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fc1b8ca505440228c8e9a544e3c549a (127.0.104.1:41509): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
W20250904 07:58:19.688262 1228 leader_election.cc:343] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45f0af8a16f14b8b83950e205831688f (127.0.104.2:44461): Not found: Tablet not found: d9c7fcc6764440609a0a5d4e6bf726bf
I20250904 07:58:19.688367 1228 leader_election.cc:304] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fce49d1a43d4cdbbfde7f0692eee50b; no voters: 2fc1b8ca505440228c8e9a544e3c549a, 45f0af8a16f14b8b83950e205831688f
I20250904 07:58:19.688580 4917 raft_consensus.cc:2747] T d9c7fcc6764440609a0a5d4e6bf726bf P 7fce49d1a43d4cdbbfde7f0692eee50b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:58:21.982298 2537 master_runner.cc:186] Error getting master registration for 127.0.104.62:34211: Network error: Client connection negotiation failed: client connection to 127.0.104.62:34211: connect: Connection refused (error 111),
I20250904 07:58:21.982483 2537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 07:58:25.904403 1346 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:58:25.904685 1346 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
W20250904 07:58:25.904902 1346 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
F20250904 07:58:25.915611 1375 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1756972705 (unix time) try "date -d @1756972705" if you are using GNU date ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status:
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000001a0) received by PID 416 (TID 0x7fef69107700) from PID 416; stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922204 1355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f9daeb5d9b94e808d296ebebcae9e9d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922204 1355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f9daeb5d9b94e808d296ebebcae9e9d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922498 1374 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922204 1355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f9daeb5d9b94e808d296ebebcae9e9d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922498 1374 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922649 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28be2f0ae2d54f5bb0b09107f6069cc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922204 1355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f9daeb5d9b94e808d296ebebcae9e9d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922498 1374 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922649 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28be2f0ae2d54f5bb0b09107f6069cc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922699 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcff30abce7b454b827a08daf85eab8d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922204 1355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f9daeb5d9b94e808d296ebebcae9e9d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922498 1374 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922649 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28be2f0ae2d54f5bb0b09107f6069cc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922699 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcff30abce7b454b827a08daf85eab8d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.923060 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2fcb873f751947deb8f788e89007069c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922204 1355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f9daeb5d9b94e808d296ebebcae9e9d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922498 1374 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922649 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28be2f0ae2d54f5bb0b09107f6069cc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922699 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcff30abce7b454b827a08daf85eab8d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.923060 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2fcb873f751947deb8f788e89007069c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.923846 1373 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fef86b79980 (unknown) at ??:0
F20250904 07:58:25.915872 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c308554984c24f29b70e853b6e48516b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.916930 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919292 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ad93fa23e5724bf3aaa93e872f55d612 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919381 1353 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da1c79fbee874fd58c1a2dfea10b89c6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919672 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5d14da275d34445abdb109af99d1cc7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1363 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.919850 1364 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.920931 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e71ba3eed2d5429796154c26d9f1c978 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922204 1355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f9daeb5d9b94e808d296ebebcae9e9d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922498 1374 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922649 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28be2f0ae2d54f5bb0b09107f6069cc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.922699 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcff30abce7b454b827a08daf85eab8d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.923060 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2fcb873f751947deb8f788e89007069c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.923846 1373 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)F20250904 07:58:25.924070 1358 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aa64673a1d5e47bdb0b862084daac8ca on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:34211,127.0.104.61:39659,127.0.104.60:40359, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40359: connect: Connection refused (error 111)
*** Check failure stack trace: ***
W20250904 07:58:25.924950 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.60:40359)
@ 0x7fef7fb3ffb7 gsignal at ??:0
@ 0x7fef7fb41921 abort at ??:0
@ 0x7fef82877dcd google::LogMessage::Fail() at ??:0
@ 0x7fef8287bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fef828777cc google::LogMessage::Flush() at ??:0
@ 0x7fef82878f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40d9f5 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:391
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fef801c76df (unknown) at ??:0
@ 0x7fef86b6e6db start_thread at ??:0
@ 0x7fef7fc2271f clone at ??:0