Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260929 15:35:05.145249 400 test_util.cc:268] Using random seed: -314609114
2026-09-29T15:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:35:05Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYukZNI/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYukZNI/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/security.properties
2026-09-29 15:35:13: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYukZNI/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYukZNI/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 15:35:21.703645 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.62:41769
--webserver_interface=127.0.100.62
--webserver_port=0
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260929 15:35:22.154410 644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:22.155194 644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:22.155759 644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:35:22.190156 644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:35:22.190514 644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:22.190810 644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:35:22.191058 644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:35:22.227020 644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.62:41769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.100.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:22.228397 644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:22.230189 644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:35:22.265403 653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:22.265868 650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:22.266364 651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:23.588380 652 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260929 15:35:23.588493 644 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260929 15:35:23.592864 644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:23.597357 644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:23.598840 644 hybrid_clock.cc:648] HybridClock initialized: now 1790696123598807 us; error 62 us; skew 500 ppm
I20260929 15:35:23.599718 644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:23.615108 644 webserver.cc:460] Webserver started at http://127.0.100.62:33553/ using document root <none> and password file <none>
I20260929 15:35:23.616202 644 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:23.616442 644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:23.616875 644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:35:23.622646 644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data/instance:
uuid: "db89c29300224c4698b6cd8be3dd3cbc"
format_stamp: "Formatted at 2026-09-29 15:35:23 on dist-test-slave-0wmn"
I20260929 15:35:23.623876 644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/wal/instance:
uuid: "db89c29300224c4698b6cd8be3dd3cbc"
format_stamp: "Formatted at 2026-09-29 15:35:23 on dist-test-slave-0wmn"
I20260929 15:35:23.633893 644 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20260929 15:35:23.644443 660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:23.645948 644 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:35:23.646373 644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/wal
uuid: "db89c29300224c4698b6cd8be3dd3cbc"
format_stamp: "Formatted at 2026-09-29 15:35:23 on dist-test-slave-0wmn"
I20260929 15:35:23.646806 644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:23.695130 644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:23.697666 644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:23.698261 644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:23.893218 644 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:41769
I20260929 15:35:23.893465 711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:41769 every 8 connection(s)
I20260929 15:35:23.896797 644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/data/info.pb
I20260929 15:35:23.906232 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 644
I20260929 15:35:23.906860 400 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-0/wal/instance
I20260929 15:35:23.907307 712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:23.914871 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.61:43161
--webserver_interface=127.0.100.61
--webserver_port=0
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260929 15:35:23.925472 712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:35:23.972375 712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
W20260929 15:35:23.977073 664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:35:23.986492 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.61:43161: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:24.036175 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } attempt: 1
W20260929 15:35:24.042690 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.61:43161: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:24.111274 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } attempt: 2
W20260929 15:35:24.116967 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.61:43161: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:24.208544 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } attempt: 3
W20260929 15:35:24.214269 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.61:43161: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:24.357784 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } attempt: 4
W20260929 15:35:24.363505 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.61:43161: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:35:24.381314 713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:24.382146 713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:24.382637 713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:35:24.414542 713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:35:24.414924 713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:24.415203 713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:35:24.415463 713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:35:24.451116 713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.61:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.100.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:24.452510 713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:24.454290 713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:35:24.490418 722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:24.663028 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } attempt: 5
W20260929 15:35:24.668958 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.61:43161: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:25.216650 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } attempt: 6
W20260929 15:35:25.229017 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.61:43161: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:35:24.490880 723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:24.496692 713 server_base.cc:1034] running on GCE node
W20260929 15:35:24.492373 725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:25.823746 713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:25.828231 713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:25.829698 713 hybrid_clock.cc:648] HybridClock initialized: now 1790696125829653 us; error 60 us; skew 500 ppm
I20260929 15:35:25.830626 713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:25.841150 713 webserver.cc:460] Webserver started at http://127.0.100.61:36561/ using document root <none> and password file <none>
I20260929 15:35:25.842293 713 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:25.842552 713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:25.843075 713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:35:25.848815 713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data/instance:
uuid: "4211c7490eeb482dafc0d5e1dcbd5506"
format_stamp: "Formatted at 2026-09-29 15:35:25 on dist-test-slave-0wmn"
I20260929 15:35:25.850171 713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/wal/instance:
uuid: "4211c7490eeb482dafc0d5e1dcbd5506"
format_stamp: "Formatted at 2026-09-29 15:35:25 on dist-test-slave-0wmn"
I20260929 15:35:25.860082 713 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260929 15:35:25.868880 735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:25.870292 713 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.004s
I20260929 15:35:25.870609 713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/wal
uuid: "4211c7490eeb482dafc0d5e1dcbd5506"
format_stamp: "Formatted at 2026-09-29 15:35:25 on dist-test-slave-0wmn"
I20260929 15:35:25.870963 713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:25.934746 713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:25.936718 713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:25.937328 713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:26.125986 713 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.61:43161
I20260929 15:35:26.126223 786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.61:43161 every 8 connection(s)
I20260929 15:35:26.129034 713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/data/info.pb
I20260929 15:35:26.133626 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 713
I20260929 15:35:26.134464 400 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-1/wal/instance
I20260929 15:35:26.137794 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.60:37099
--webserver_interface=127.0.100.60
--webserver_port=0
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260929 15:35:26.138136 787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:26.155759 787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:35:26.192224 787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:35:26.220415 787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
W20260929 15:35:26.223654 739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
W20260929 15:35:26.232699 787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.282368 787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 1
W20260929 15:35:26.288404 787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.289551 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } attempt: 7
I20260929 15:35:26.313041 712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
W20260929 15:35:26.323463 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.356948 787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 2
W20260929 15:35:26.363040 787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.382025 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 1
W20260929 15:35:26.388721 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.454663 787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 3
W20260929 15:35:26.460793 787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.470283 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 2
W20260929 15:35:26.475838 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.561388 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 3
W20260929 15:35:26.566859 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.604261 787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 4
W20260929 15:35:26.609854 787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
W20260929 15:35:26.632977 788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:26.633746 788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:26.634227 788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:35:26.665791 788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:35:26.666175 788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:26.666469 788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:35:26.666726 788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:35:26.703508 788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.60:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.100.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:26.704907 788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:26.706562 788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:35:26.707408 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 4
W20260929 15:35:26.712406 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
W20260929 15:35:26.735944 800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:26.909371 787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 5
W20260929 15:35:26.914630 787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:26.995973 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 5
W20260929 15:35:27.002286 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:27.462457 787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 6
W20260929 15:35:27.476857 787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:27.554772 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 6
W20260929 15:35:27.569104 712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.100.60:37099: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
W20260929 15:35:26.736383 801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:26.740418 803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:26.741919 788 server_base.cc:1034] running on GCE node
I20260929 15:35:28.125061 788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:28.129410 788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:28.130960 788 hybrid_clock.cc:648] HybridClock initialized: now 1790696128130908 us; error 74 us; skew 500 ppm
I20260929 15:35:28.131999 788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:28.145227 788 webserver.cc:460] Webserver started at http://127.0.100.60:36831/ using document root <none> and password file <none>
I20260929 15:35:28.146380 788 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:28.146677 788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:28.147193 788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:35:28.153733 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/instance:
uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
format_stamp: "Formatted at 2026-09-29 15:35:28 on dist-test-slave-0wmn"
I20260929 15:35:28.155074 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal/instance:
uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
format_stamp: "Formatted at 2026-09-29 15:35:28 on dist-test-slave-0wmn"
I20260929 15:35:28.170066 788 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.006s sys 0.009s
I20260929 15:35:28.181046 814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:28.182675 788 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260929 15:35:28.183077 788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
format_stamp: "Formatted at 2026-09-29 15:35:28 on dist-test-slave-0wmn"
I20260929 15:35:28.183532 788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:28.237846 788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:28.239935 788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:28.240556 788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:28.432808 788 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.60:37099
I20260929 15:35:28.433020 865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.60:37099 every 8 connection(s)
I20260929 15:35:28.435977 788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
I20260929 15:35:28.440002 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 788
I20260929 15:35:28.440400 400 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal/instance
I20260929 15:35:28.445291 866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:28.480588 866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:35:28.526401 866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:35:28.537335 787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 7
I20260929 15:35:28.562052 866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:35:28.589618 787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Bootstrap starting.
I20260929 15:35:28.592368 866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Bootstrap starting.
I20260929 15:35:28.602245 787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:28.602672 712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } attempt: 7
I20260929 15:35:28.607810 866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:28.607633 787 log.cc:824] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:28.613384 866 log.cc:824] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:28.623777 787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: No bootstrap required, opened a new log
I20260929 15:35:28.627388 866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: No bootstrap required, opened a new log
I20260929 15:35:28.656833 712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: Bootstrap starting.
I20260929 15:35:28.672477 712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:28.673363 866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:28.674697 866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:28.675179 866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02435ab47dcc45cd9b1b9b37bb1dc178, State: Initialized, Role: FOLLOWER
I20260929 15:35:28.676719 866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:28.678246 712 log.cc:824] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:28.680300 787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:28.682322 787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:28.683034 787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4211c7490eeb482dafc0d5e1dcbd5506, State: Initialized, Role: FOLLOWER
I20260929 15:35:28.682415 878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:28.683779 878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:28.685063 787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:28.685956 866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:35:28.693583 712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: No bootstrap required, opened a new log
I20260929 15:35:28.704308 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:28.705965 787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:35:28.706360 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:28.750097 712 raft_consensus.cc:348] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
W20260929 15:35:28.751636 820 tablet.cc:2369] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:35:28.751885 712 raft_consensus.cc:374] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:28.752462 712 raft_consensus.cc:729] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89c29300224c4698b6cd8be3dd3cbc, State: Initialized, Role: FOLLOWER
I20260929 15:35:28.754567 712 consensus_queue.cc:260] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:28.783334 712 sys_catalog.cc:564] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:35:28.787695 897 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:28.789608 897 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:28.832401 906 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:35:28.839205 788 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: OK, instance_id { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" instance_seqno: 1790696123827505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:35:28.915220 788 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:35:28.951647 741 tablet.cc:2369] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:35:28.969403 919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:35:28.969251 666 tablet.cc:2369] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:35:28.970093 919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:35:28.937551 713 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: OK, instance_id { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" instance_seqno: 1790696123827505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:35:29.012104 713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:35:29.059520 644 master_runner.cc:263] No leader master found from master db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:29.059969 644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:35:29.062071 897 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:29.062732 897 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
W20260929 15:35:29.063323 927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:35:29.063853 927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:35:29.066895 897 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:29.070791 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:29.073943 762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:29.075635 841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89c29300224c4698b6cd8be3dd3cbc in term 0.
I20260929 15:35:29.076879 762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89c29300224c4698b6cd8be3dd3cbc in term 0.
I20260929 15:35:29.079592 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178, db89c29300224c4698b6cd8be3dd3cbc; no voters:
I20260929 15:35:29.082614 897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:35:29.083323 897 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:35:29.083952 897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:29.098142 897 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:29.102555 762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506"
I20260929 15:35:29.103364 762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:29.117089 762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89c29300224c4698b6cd8be3dd3cbc in term 1.
I20260929 15:35:29.120172 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
I20260929 15:35:29.121070 841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:29.130002 897 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 1 election: Requested vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:29.146646 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc; no voters:
I20260929 15:35:29.148159 897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:35:29.154461 841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89c29300224c4698b6cd8be3dd3cbc in term 1.
I20260929 15:35:29.160670 897 raft_consensus.cc:686] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 1 LEADER]: Becoming Leader. State: Replica: db89c29300224c4698b6cd8be3dd3cbc, State: Running, Role: LEADER
I20260929 15:35:29.162379 897 consensus_queue.cc:237] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:29.191473 929 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader db89c29300224c4698b6cd8be3dd3cbc. Latest consensus state: current_term: 1 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:29.192564 929 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:29.205581 931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:35:29.219416 931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:35:29.259217 762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 1 FOLLOWER]: Refusing update from remote peer db89c29300224c4698b6cd8be3dd3cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:35:29.260593 841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Refusing update from remote peer db89c29300224c4698b6cd8be3dd3cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:35:29.262141 929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 15:35:29.264856 929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 15:35:29.314329 934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db89c29300224c4698b6cd8be3dd3cbc. Latest consensus state: current_term: 1 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:29.319418 934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:29.327344 929 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:29.328563 929 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:29.332772 935 mvcc.cc:205] Tried to move back new op lower bound from 7334691345415663616 to 7334691345066635264. Current Snapshot: MvccSnapshot[applied={T|T < 7334691345415663616}]
I20260929 15:35:29.335413 933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db89c29300224c4698b6cd8be3dd3cbc. Latest consensus state: current_term: 1 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:29.336588 933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:29.345346 931 catalog_manager.cc:1347] Generated new cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:29.345817 931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:35:29.345465 897 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:29.346530 897 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:29.352650 934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:29.353884 934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:29.367108 933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:29.368285 933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:29.412824 931 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:35:29.416997 931 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:35:29.530274 931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: Generated new TSK 0
I20260929 15:35:29.532295 931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:35:29.541247 931 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260929 15:35:29.550699 931 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260929 15:35:29.582046 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.100.1:0
--local_ip_for_outbound_sockets=127.0.100.1
--webserver_interface=127.0.100.1
--webserver_port=0
--tserver_master_addrs=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260929 15:35:29.865410 906 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:29.866279 906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: loading cluster ID for follower catalog manager: success
I20260929 15:35:29.910317 906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: acquiring CA information for follower catalog manager: success
I20260929 15:35:29.936380 906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:35:29.978089 919 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:29.978433 919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: loading cluster ID for follower catalog manager: success
I20260929 15:35:29.996021 919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: acquiring CA information for follower catalog manager: success
I20260929 15:35:30.016676 919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 15:35:30.331746 943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:30.332741 943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:30.333910 943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:35:30.396116 943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:30.397964 943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.100.1
I20260929 15:35:30.413532 949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:30.435812 949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:30.441185 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:30.452543 949 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:30.454226 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:30.458644 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:30.465690 949 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:35:30.483961 943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.100.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.100.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.100.1
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:30.486469 943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:30.489130 943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:35:30.495883 950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:30.498003 950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:30.512916 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:30.514400 841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4211c7490eeb482dafc0d5e1dcbd5506 in term 1.
I20260929 15:35:30.516860 739 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506; no voters:
I20260929 15:35:30.521190 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:30.546607 950 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:30.547703 950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 15:35:30.548360 950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 1 FOLLOWER]: Starting leader election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:30.549021 950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 1 FOLLOWER]: Advancing to term 2
W20260929 15:35:30.558357 954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:30.571817 739 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.100.60:37099, user_credentials={real_user=slave}} blocked reactor thread for 55175.8us
I20260929 15:35:30.595801 950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:30.612138 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc"
I20260929 15:35:30.617170 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
I20260929 15:35:30.619824 739 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:30.621731 955 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 15:35:30.624641 950 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 2 election: Requested vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:31.143816 761 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Rejecting Update request from peer db89c29300224c4698b6cd8be3dd3cbc for earlier term 1. Current term is 2. Ops: []
I20260929 15:35:31.146811 953 consensus_queue.cc:1046] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 15:35:31.166157 953 raft_consensus.cc:3032] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 1 LEADER]: Stepping down as leader of term 1
I20260929 15:35:31.167162 953 raft_consensus.cc:729] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: db89c29300224c4698b6cd8be3dd3cbc, State: Running, Role: LEADER
I20260929 15:35:31.169021 953 consensus_queue.cc:260] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:31.172473 953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:35:31.366896 963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:31.371052 963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:31.382068 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:31.383291 687 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4211c7490eeb482dafc0d5e1dcbd5506 in term 2.
I20260929 15:35:31.385926 738 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc; no voters:
I20260929 15:35:31.394052 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:31.400767 963 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:31.403029 963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 15:35:31.405784 963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:35:31.406481 963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:35:31.430814 963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:31.448971 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc"
I20260929 15:35:31.449998 687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:35:31.461512 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
I20260929 15:35:31.464548 963 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 3 election: Requested vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:31.473302 687 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4211c7490eeb482dafc0d5e1dcbd5506 in term 3.
I20260929 15:35:31.479737 738 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178
I20260929 15:35:31.481958 963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 15:35:31.483227 963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 3 LEADER]: Becoming Leader. State: Replica: 4211c7490eeb482dafc0d5e1dcbd5506, State: Running, Role: LEADER
I20260929 15:35:31.485644 963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:31.509176 964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4211c7490eeb482dafc0d5e1dcbd5506. Latest consensus state: current_term: 3 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:31.511502 964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:31.521670 966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:35:31.555326 966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:35:31.564886 966 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:31.565449 966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:35:31.576387 966 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:35:31.577907 967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:31.578680 967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:31.584031 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:31.585584 687 raft_consensus.cc:2354] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02435ab47dcc45cd9b1b9b37bb1dc178 for earlier term 2. Current term is 3.
I20260929 15:35:31.585650 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:31.587885 967 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:31.588744 817 leader_election.cc:398] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 2 pre-election: Vote denied by peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02435ab47dcc45cd9b1b9b37bb1dc178 for earlier term 2. Current term is 3.
I20260929 15:35:31.589280 817 leader_election.cc:401] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260929 15:35:31.594259 967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 1 FOLLOWER]: Advancing to term 3
I20260929 15:35:31.608968 966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Loaded TSK: 0
I20260929 15:35:31.611729 966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:35:31.616474 967 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02435ab47dcc45cd9b1b9b37bb1dc178 for earlier term 2. Current term is 3.
I20260929 15:35:31.632422 966 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260929 15:35:31.640136 966 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260929 15:35:31.917647 841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Refusing update from remote peer 4211c7490eeb482dafc0d5e1dcbd5506: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 15:35:31.920730 964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 15:35:31.939733 687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Refusing update from remote peer 4211c7490eeb482dafc0d5e1dcbd5506: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 15:35:31.942018 964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 15:35:31.949090 967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4211c7490eeb482dafc0d5e1dcbd5506. Latest consensus state: current_term: 3 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:31.950090 967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:31.967342 963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:31.968537 963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:31.968340 969 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4211c7490eeb482dafc0d5e1dcbd5506. Latest consensus state: current_term: 3 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:31.969771 969 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:31.982188 973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:31.986035 973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:31.989617 967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:31.990707 967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:31.995463 969 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:31.996428 969 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:32.075754 927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:35:32.858886 976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:32.859727 976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:32.864746 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:32.872017 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:32.874512 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:32.882403 976 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:32.883890 976 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 15:35:31.921094 952 debug-util.cc:398] Leaking SignalData structure 0x7b08000148c0 after lost signal to thread 943
W20260929 15:35:33.470445 952 kernel_stack_watchdog.cc:198] Thread 943 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:637 for 330ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260929 15:35:30.594465 956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:33.477183 959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:33.473978 943 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 2.887s user 0.456s sys 1.035s
W20260929 15:35:33.480019 943 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 2.893s user 0.456s sys 1.035s
W20260929 15:35:33.488679 958 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260929 15:35:33.489388 943 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260929 15:35:33.490908 943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:33.504685 943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:33.506237 943 hybrid_clock.cc:648] HybridClock initialized: now 1790696133506198 us; error 48 us; skew 500 ppm
I20260929 15:35:33.507664 943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:33.532089 943 webserver.cc:460] Webserver started at http://127.0.100.1:37273/ using document root <none> and password file <none>
I20260929 15:35:33.533761 943 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:33.534148 943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:33.535009 943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:35:33.543017 943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data/instance:
uuid: "8f323e4c5c0f4d63bd2de77984299c86"
format_stamp: "Formatted at 2026-09-29 15:35:33 on dist-test-slave-0wmn"
I20260929 15:35:33.544843 943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal/instance:
uuid: "8f323e4c5c0f4d63bd2de77984299c86"
format_stamp: "Formatted at 2026-09-29 15:35:33 on dist-test-slave-0wmn"
I20260929 15:35:33.568840 943 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.007s sys 0.011s
I20260929 15:35:33.585855 982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:33.587962 943 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.008s
I20260929 15:35:33.588512 943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal
uuid: "8f323e4c5c0f4d63bd2de77984299c86"
format_stamp: "Formatted at 2026-09-29 15:35:33 on dist-test-slave-0wmn"
I20260929 15:35:33.589177 943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:33.675714 943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:33.678877 943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:33.679751 943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:33.686740 943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:35:33.696751 943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:35:33.697139 943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.001s
I20260929 15:35:33.697563 943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:35:33.697831 943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:34.396683 943 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:45865
I20260929 15:35:34.396833 1095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:45865 every 8 connection(s)
I20260929 15:35:34.400760 943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/data/info.pb
I20260929 15:35:34.409356 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 943
I20260929 15:35:34.409863 400 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal/instance
I20260929 15:35:34.424787 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.100.2:0
--local_ip_for_outbound_sockets=127.0.100.2
--webserver_interface=127.0.100.2
--webserver_port=0
--tserver_master_addrs=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260929 15:35:34.512359 1099 heartbeater.cc:347] Connected to a master server at 127.0.100.62:41769
I20260929 15:35:34.512876 1099 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:34.514081 1099 heartbeater.cc:511] Master 127.0.100.62:41769 requested a full tablet report, sending...
I20260929 15:35:34.517705 677 ts_manager.cc:194] Registered new tserver with Master: 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865)
I20260929 15:35:34.519102 1096 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:34.519459 1096 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:34.520342 1096 heartbeater.cc:511] Master 127.0.100.60:37099 requested a full tablet report, sending...
I20260929 15:35:34.523950 1097 heartbeater.cc:347] Connected to a master server at 127.0.100.61:43161
I20260929 15:35:34.524318 1097 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:34.524333 831 ts_manager.cc:194] Registered new tserver with Master: 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865)
I20260929 15:35:34.525208 1097 heartbeater.cc:511] Master 127.0.100.61:43161 requested a full tablet report, sending...
I20260929 15:35:34.527918 752 ts_manager.cc:194] Registered new tserver with Master: 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865)
I20260929 15:35:34.532418 752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.100.1:56637
I20260929 15:35:34.758937 1107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:34.759600 1107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:34.762737 1107 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:34.764300 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:34.765745 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:34.768442 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:34.769960 1107 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 15:35:35.012240 1106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:35.012902 1106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:35.013538 1106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:35:35.037508 1097 heartbeater.cc:503] Master 127.0.100.61:43161 was elected leader, sending a full tablet report...
W20260929 15:35:35.048658 1106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:35.049708 1106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.100.2
I20260929 15:35:35.085906 1106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.100.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.100.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.100.2
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:35.087555 1106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:35.089475 1106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:35:35.110468 1116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:35.827437 1120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:35.827867 1120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:35.829335 1120 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:35.830518 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:35.831167 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:35.832656 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:35.833295 1120 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 15:35:36.443094 1121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:36.443739 1121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:36.446453 1121 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:36.447996 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:36.448910 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:36.450747 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:36.452440 1121 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 15:35:35.112026 1113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:35.110492 1114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:36.526793 1115 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260929 15:35:36.526947 1106 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260929 15:35:36.531553 1106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:36.534754 1106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:36.536322 1106 hybrid_clock.cc:648] HybridClock initialized: now 1790696136536272 us; error 48 us; skew 500 ppm
I20260929 15:35:36.537425 1106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:36.549187 1106 webserver.cc:460] Webserver started at http://127.0.100.2:36827/ using document root <none> and password file <none>
I20260929 15:35:36.550387 1106 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:36.550663 1106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:36.551184 1106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:35:36.557723 1106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data/instance:
uuid: "3d101b58a89c4d418feb4c9a65cbd21c"
format_stamp: "Formatted at 2026-09-29 15:35:36 on dist-test-slave-0wmn"
I20260929 15:35:36.559181 1106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal/instance:
uuid: "3d101b58a89c4d418feb4c9a65cbd21c"
format_stamp: "Formatted at 2026-09-29 15:35:36 on dist-test-slave-0wmn"
I20260929 15:35:36.569448 1106 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.008s
I20260929 15:35:36.577991 1126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:36.579272 1106 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.008s
I20260929 15:35:36.579702 1106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal
uuid: "3d101b58a89c4d418feb4c9a65cbd21c"
format_stamp: "Formatted at 2026-09-29 15:35:36 on dist-test-slave-0wmn"
I20260929 15:35:36.580107 1106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:36.627552 1106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:36.629175 1106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:36.629729 1106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:36.632673 1106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:35:36.637295 1106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:35:36.637573 1106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 15:35:36.637849 1106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:35:36.638026 1106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:36.903060 1106 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:46845
I20260929 15:35:36.903509 1239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:46845 every 8 connection(s)
I20260929 15:35:36.906309 1106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/data/info.pb
I20260929 15:35:36.914304 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 1106
I20260929 15:35:36.915509 400 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal/instance
I20260929 15:35:36.949438 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.100.3:0
--local_ip_for_outbound_sockets=127.0.100.3
--webserver_interface=127.0.100.3
--webserver_port=0
--tserver_master_addrs=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260929 15:35:37.006387 1250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:37.008235 1250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:37.014988 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:37.023988 1250 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:37.024909 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:37.027968 817 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:37.029385 1250 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 15:35:37.034698 1240 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:37.035413 1240 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:37.037488 1240 heartbeater.cc:511] Master 127.0.100.60:37099 requested a full tablet report, sending...
I20260929 15:35:37.037711 1242 heartbeater.cc:347] Connected to a master server at 127.0.100.62:41769
I20260929 15:35:37.038192 1242 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:37.039631 1242 heartbeater.cc:511] Master 127.0.100.62:41769 requested a full tablet report, sending...
I20260929 15:35:37.044265 831 ts_manager.cc:194] Registered new tserver with Master: 3d101b58a89c4d418feb4c9a65cbd21c (127.0.100.2:46845)
I20260929 15:35:37.044461 677 ts_manager.cc:194] Registered new tserver with Master: 3d101b58a89c4d418feb4c9a65cbd21c (127.0.100.2:46845)
I20260929 15:35:37.049773 1241 heartbeater.cc:347] Connected to a master server at 127.0.100.61:43161
I20260929 15:35:37.050262 1241 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:37.051403 1241 heartbeater.cc:511] Master 127.0.100.61:43161 requested a full tablet report, sending...
I20260929 15:35:37.054263 752 ts_manager.cc:194] Registered new tserver with Master: 3d101b58a89c4d418feb4c9a65cbd21c (127.0.100.2:46845)
I20260929 15:35:37.057200 752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.100.2:38981
I20260929 15:35:37.544906 1256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:37.545451 1256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:37.547101 1256 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:37.547981 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:37.549559 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:37.552568 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:37.553366 1256 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 15:35:37.554858 1251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:37.555325 1251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:37.555861 1251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:35:37.564103 1241 heartbeater.cc:503] Master 127.0.100.61:43161 was elected leader, sending a full tablet report...
W20260929 15:35:37.588742 1251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:37.589833 1251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.100.3
I20260929 15:35:37.626202 1251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.100.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.100.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.100.3
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:37.627563 1251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:37.629706 1251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:35:37.640789 1259 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:37.641480 1259 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:37.644649 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:37.646701 1259 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:37.649418 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:37.652048 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db89c29300224c4698b6cd8be3dd3cbc; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506
I20260929 15:35:37.653292 1259 raft_consensus.cc:2726] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 15:35:37.666364 1262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:38.132131 1265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:38.132565 1265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:38.135146 1265 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:38.136201 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:38.136914 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:38.139045 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:38.140205 1265 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 15:35:38.254308 1266 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:38.254729 1266 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:38.256304 1266 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:38.257596 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:38.257731 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:38.259514 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db89c29300224c4698b6cd8be3dd3cbc; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506
I20260929 15:35:38.260304 1266 raft_consensus.cc:2726] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 15:35:38.747017 1267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:38.747814 1267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:38.750492 1267 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:38.751870 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:38.752692 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:38.754459 817 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:38.755434 1267 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 15:35:38.774013 1268 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:38.774531 1268 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:38.776635 1268 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:38.777812 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:38.777881 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:38.779459 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db89c29300224c4698b6cd8be3dd3cbc; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506
I20260929 15:35:38.780090 1268 raft_consensus.cc:2726] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 15:35:37.666896 1260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:37.666831 1258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:39.035580 1261 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260929 15:35:39.035980 1251 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260929 15:35:39.040259 1251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:39.043409 1251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:39.044826 1251 hybrid_clock.cc:648] HybridClock initialized: now 1790696139044783 us; error 49 us; skew 500 ppm
I20260929 15:35:39.045881 1251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:39.062718 1251 webserver.cc:460] Webserver started at http://127.0.100.3:43861/ using document root <none> and password file <none>
I20260929 15:35:39.064410 1251 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:39.064864 1251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:39.065847 1251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:35:39.072831 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data/instance:
uuid: "20e5d3d83abe4b5681aeac687d0ae209"
format_stamp: "Formatted at 2026-09-29 15:35:39 on dist-test-slave-0wmn"
I20260929 15:35:39.074213 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal/instance:
uuid: "20e5d3d83abe4b5681aeac687d0ae209"
format_stamp: "Formatted at 2026-09-29 15:35:39 on dist-test-slave-0wmn"
I20260929 15:35:39.085947 1251 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.005s
I20260929 15:35:39.094784 1273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:39.096011 1251 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260929 15:35:39.096422 1251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal
uuid: "20e5d3d83abe4b5681aeac687d0ae209"
format_stamp: "Formatted at 2026-09-29 15:35:39 on dist-test-slave-0wmn"
I20260929 15:35:39.096761 1251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:39.142132 1251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:39.144326 1251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:39.144958 1251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:39.152041 1251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:35:39.159657 1251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:35:39.159925 1251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:39.160197 1251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:35:39.160372 1251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:39.261098 1302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:39.262280 1302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.267846 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:39.268965 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:39.269724 1302 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:39.271835 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:39.273455 1302 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 15:35:39.315295 1318 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:39.316033 1318 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.319135 1318 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:39.320482 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:39.321053 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:39.321871 841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89c29300224c4698b6cd8be3dd3cbc in term 3.
I20260929 15:35:39.323115 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178, db89c29300224c4698b6cd8be3dd3cbc; no voters: 4211c7490eeb482dafc0d5e1dcbd5506
I20260929 15:35:39.323868 1318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 15:35:39.324275 1318 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:39.324672 1318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 3 FOLLOWER]: Advancing to term 4
I20260929 15:35:39.336246 1318 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.338949 1318 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 election: Requested vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:39.340466 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506"
I20260929 15:35:39.340700 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
I20260929 15:35:39.343627 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db89c29300224c4698b6cd8be3dd3cbc; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506
I20260929 15:35:39.345583 1318 raft_consensus.cc:2726] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 15:35:39.358142 687 raft_consensus.cc:1226] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Rejecting Update request from peer 4211c7490eeb482dafc0d5e1dcbd5506 for earlier term 3. Current term is 4. Ops: []
I20260929 15:35:39.361814 1322 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260929 15:35:39.365778 1331 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 15:35:39.366761 1331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4211c7490eeb482dafc0d5e1dcbd5506, State: Running, Role: LEADER
I20260929 15:35:39.368846 1331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.372284 1331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 3 FOLLOWER]: Advancing to term 4
I20260929 15:35:39.598812 1251 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:36189
I20260929 15:35:39.599134 1389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:36189 every 8 connection(s)
I20260929 15:35:39.601958 1251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/data/info.pb
I20260929 15:35:39.603636 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 1251
I20260929 15:35:39.604060 400 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal/instance
I20260929 15:35:39.699326 1392 heartbeater.cc:347] Connected to a master server at 127.0.100.62:41769
I20260929 15:35:39.699990 1392 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:39.701757 1392 heartbeater.cc:511] Master 127.0.100.62:41769 requested a full tablet report, sending...
I20260929 15:35:39.702136 1390 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:39.702759 1390 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:39.704154 1390 heartbeater.cc:511] Master 127.0.100.60:37099 requested a full tablet report, sending...
I20260929 15:35:39.707528 677 ts_manager.cc:194] Registered new tserver with Master: 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:39.707633 1391 heartbeater.cc:347] Connected to a master server at 127.0.100.61:43161
I20260929 15:35:39.708112 1391 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:39.708472 831 ts_manager.cc:194] Registered new tserver with Master: 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:39.709494 1391 heartbeater.cc:511] Master 127.0.100.61:43161 requested a full tablet report, sending...
I20260929 15:35:39.712494 752 ts_manager.cc:194] Registered new tserver with Master: 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:39.733558 400 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:35:39.834393 1407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:39.835124 1407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.837046 1407 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:39.838168 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
I20260929 15:35:39.838480 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:39.839349 687 raft_consensus.cc:2379] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02435ab47dcc45cd9b1b9b37bb1dc178 in current term 4: Already voted for candidate db89c29300224c4698b6cd8be3dd3cbc in this term.
I20260929 15:35:39.839334 761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02435ab47dcc45cd9b1b9b37bb1dc178 in term 4.
I20260929 15:35:39.840713 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506; no voters:
I20260929 15:35:39.841713 1407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 15:35:39.842233 1407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:39.842641 1407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 3 FOLLOWER]: Advancing to term 4
I20260929 15:35:39.852017 1407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.854837 1407 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 election: Requested vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:39.854825 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc"
I20260929 15:35:39.855052 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506"
I20260929 15:35:39.855669 687 raft_consensus.cc:2379] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02435ab47dcc45cd9b1b9b37bb1dc178 in current term 4: Already voted for candidate db89c29300224c4698b6cd8be3dd3cbc in this term.
I20260929 15:35:39.860991 761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02435ab47dcc45cd9b1b9b37bb1dc178 in term 4.
I20260929 15:35:39.862306 818 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506; no voters: db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:39.863169 1407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 15:35:39.864674 1407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 LEADER]: Becoming Leader. State: Replica: 02435ab47dcc45cd9b1b9b37bb1dc178, State: Running, Role: LEADER
I20260929 15:35:39.865921 1407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.870815 1408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02435ab47dcc45cd9b1b9b37bb1dc178. Latest consensus state: current_term: 4 leader_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:39.871691 1408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:39.874601 1410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:35:39.880393 1410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:35:39.884073 1410 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:39.884332 1410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:35:39.889377 1410 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:35:39.893568 1410 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Loaded TSK: 0
I20260929 15:35:39.895339 1410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:35:39.899731 1410 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260929 15:35:39.904646 1410 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260929 15:35:39.921222 1411 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:39.921849 1411 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.924366 1411 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:39.925169 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:39.925496 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:39.925962 761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89c29300224c4698b6cd8be3dd3cbc in term 4.
I20260929 15:35:39.927567 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178
I20260929 15:35:39.928478 1411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 15:35:39.928798 1411 raft_consensus.cc:482] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:35:39.929159 1411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 4 FOLLOWER]: Advancing to term 5
I20260929 15:35:39.940101 1411 raft_consensus.cc:504] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.942924 1411 leader_election.cc:288] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 5 election: Requested vote from peers 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:39.944617 841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
I20260929 15:35:39.945091 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db89c29300224c4698b6cd8be3dd3cbc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506"
I20260929 15:35:39.945994 761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 4 FOLLOWER]: Advancing to term 5
I20260929 15:35:39.952494 831 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53392:
name: "default.Table_e96a2bf8603447e6a2d4b4558b4ac762"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:35:39.955150 830 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53392:
name: "default.Table_660e1e2557514985886f76f5f9fe0a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:35:39.962464 761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89c29300224c4698b6cd8be3dd3cbc in term 5.
I20260929 15:35:39.964282 664 leader_election.cc:302] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178
I20260929 15:35:39.965739 1411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 5 FOLLOWER]: Leader election won for term 5
I20260929 15:35:39.971941 829 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53392:
name: "default.Table_7235c1c5685042129526757568b55d04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:35:39.977469 831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e96a2bf8603447e6a2d4b4558b4ac762 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:35:39.978107 1411 raft_consensus.cc:686] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [term 5 LEADER]: Becoming Leader. State: Replica: db89c29300224c4698b6cd8be3dd3cbc, State: Running, Role: LEADER
I20260929 15:35:39.979344 1411 consensus_queue.cc:237] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:39.991418 828 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53392:
name: "default.Table_95792767565c4269a5550bf589d8a379"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:35:39.992420 830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_660e1e2557514985886f76f5f9fe0a0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:35:40.002002 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader db89c29300224c4698b6cd8be3dd3cbc. Latest consensus state: current_term: 5 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:40.004144 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:40.008922 1422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:35:40.022529 1422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:35:40.035064 1422 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:40.035410 1422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:35:40.043457 1422 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:35:40.049556 1422 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc: Loaded TSK: 0
I20260929 15:35:40.051829 1422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:35:40.059067 1422 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260929 15:35:40.064869 1422 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260929 15:35:40.083226 400 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu with pid 788
I20260929 15:35:40.128157 677 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39896:
name: "default.Table_95792767565c4269a5550bf589d8a379"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:35:40.192068 677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_95792767565c4269a5550bf589d8a379 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:35:40.216391 676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.100.3:59843
W20260929 15:35:40.236027 1390 heartbeater.cc:644] Failed to heartbeat to 127.0.100.60:37099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
W20260929 15:35:40.448068 664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20260929 15:35:40.461697 664 consensus_peers.cc:487] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc -> Peer 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099): Couldn't send request to peer 02435ab47dcc45cd9b1b9b37bb1dc178. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:35:40.532220 761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Refusing update from remote peer db89c29300224c4698b6cd8be3dd3cbc: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260929 15:35:40.534065 1416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 15:35:40.574203 1099 heartbeater.cc:503] Master 127.0.100.62:41769 was elected leader, sending a full tablet report...
I20260929 15:35:40.577250 1437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db89c29300224c4698b6cd8be3dd3cbc. Latest consensus state: current_term: 5 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:40.580443 1437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:40.581525 1240 heartbeater.cc:503] Master 127.0.100.60:37099 was elected leader, sending a full tablet report...
I20260929 15:35:40.581645 1096 heartbeater.cc:503] Master 127.0.100.60:37099 was elected leader, sending a full tablet report...
I20260929 15:35:40.584949 1242 heartbeater.cc:503] Master 127.0.100.62:41769 was elected leader, sending a full tablet report...
I20260929 15:35:40.581101 1438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:40.598491 1438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
W20260929 15:35:40.611168 1240 heartbeater.cc:644] Failed to heartbeat to 127.0.100.60:37099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:40.616384 1438 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:40.617202 687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "db89c29300224c4698b6cd8be3dd3cbc" is_pre_election: true
W20260929 15:35:40.625614 1096 heartbeater.cc:644] Failed to heartbeat to 127.0.100.60:37099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:40.618430 1437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
W20260929 15:35:40.623613 739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260929 15:35:40.627192 1437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:40.630052 1440 sys_catalog.cc:455] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "db89c29300224c4698b6cd8be3dd3cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:40.631049 1440 sys_catalog.cc:458] T 00000000000000000000000000000000 P db89c29300224c4698b6cd8be3dd3cbc [sys.catalog]: This master's current role is: LEADER
W20260929 15:35:40.649008 739 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099): Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:40.649581 739 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:40.650355 1438 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 15:35:40.724797 1392 heartbeater.cc:503] Master 127.0.100.62:41769 was elected leader, sending a full tablet report...
I20260929 15:35:40.793944 677 catalog_manager.cc:2148] created HMS catalog entry for table [id=566074959d5b48c4b94b6df736b3eed0]
I20260929 15:35:40.825274 677 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39896:
table { table_name: "default.Table_95792767565c4269a5550bf589d8a379" } new_table_name: "default.Table_879ae5d00f944b438033d879db00571e" modify_external_catalogs: true
F20260929 15:35:40.834391 925 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_e96a2bf8603447e6a2d4b4558b4ac762
*** Check failure stack trace: ***
@ 0x7f41180bf9a0 google::LogMessage::Flush() at ??:0
I20260929 15:35:40.900594 1175 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca4ee8b9bf243339757aa47f51c0203 (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:40.900660 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca4ee8b9bf243339757aa47f51c0203 (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:40.903113 1175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca4ee8b9bf243339757aa47f51c0203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:40.903822 1030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca4ee8b9bf243339757aa47f51c0203. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f41180bf70f google::LogMessage::~LogMessage() at ??:0
I20260929 15:35:40.959653 1324 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca4ee8b9bf243339757aa47f51c0203 (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:40.962018 1324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca4ee8b9bf243339757aa47f51c0203. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f412315aaea kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260929 15:35:40.986409 1458 tablet_bootstrap.cc:492] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Bootstrap starting.
I20260929 15:35:40.996930 1458 tablet_bootstrap.cc:654] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Neither blocks nor log segments found. Creating new log.
@ 0x7f412315a33c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260929 15:35:40.999701 1460 tablet_bootstrap.cc:492] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Bootstrap starting.
I20260929 15:35:41.000526 1458 log.cc:824] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:41.017428 1460 tablet_bootstrap.cc:654] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:41.020716 1460 log.cc:824] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:41.024160 676 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39896:
name: "default.Table_fcbf632cc0894e83aa7c8dedc9ff1e58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f412315dfd2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260929 15:35:41.029438 1458 tablet_bootstrap.cc:492] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: No bootstrap required, opened a new log
I20260929 15:35:41.030349 1458 ts_tablet_manager.cc:1397] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.004s
I20260929 15:35:41.046078 1460 tablet_bootstrap.cc:492] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: No bootstrap required, opened a new log
I20260929 15:35:41.047060 1460 ts_tablet_manager.cc:1397] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Time spent bootstrapping tablet: real 0.048s user 0.015s sys 0.020s
@ 0x7f412315df8a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260929 15:35:41.061471 1466 tablet_bootstrap.cc:492] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Bootstrap starting.
@ 0x7f412315df1a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260929 15:35:41.087247 1458 raft_consensus.cc:348] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:41.088311 1458 raft_consensus.cc:374] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:41.088418 1466 tablet_bootstrap.cc:654] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:41.088629 1458 raft_consensus.cc:729] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d101b58a89c4d418feb4c9a65cbd21c, State: Initialized, Role: FOLLOWER
I20260929 15:35:41.089699 1458 consensus_queue.cc:260] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:41.090870 1466 log.cc:824] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:41.091246 1460 raft_consensus.cc:348] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:41.092296 1460 raft_consensus.cc:374] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:41.092664 1460 raft_consensus.cc:729] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f323e4c5c0f4d63bd2de77984299c86, State: Initialized, Role: FOLLOWER
I20260929 15:35:41.093950 1460 consensus_queue.cc:260] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:41.107050 1460 ts_tablet_manager.cc:1428] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Time spent starting tablet: real 0.060s user 0.046s sys 0.008s
I20260929 15:35:41.119089 1466 tablet_bootstrap.cc:492] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: No bootstrap required, opened a new log
I20260929 15:35:41.120070 1466 ts_tablet_manager.cc:1397] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.005s
@ 0x7f412315dee2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260929 15:35:41.155110 1458 ts_tablet_manager.cc:1428] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Time spent starting tablet: real 0.124s user 0.055s sys 0.043s
@ 0x7f412315d1de std::__1::__function::__func<>::operator()() at ??:0
I20260929 15:35:41.192569 1466 raft_consensus.cc:348] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
I20260929 15:35:41.194008 1466 raft_consensus.cc:374] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:41.194543 1466 raft_consensus.cc:729] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e5d3d83abe4b5681aeac687d0ae209, State: Initialized, Role: FOLLOWER
@ 0x5805ba std::__1::function<>::operator()() at ??:?
I20260929 15:35:41.196089 1466 consensus_queue.cc:260] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:41.203954 1466 ts_tablet_manager.cc:1428] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Time spent starting tablet: real 0.084s user 0.045s sys 0.007s
@ 0x7f411a270dc7 kudu::Thread::SuperviseThread() at ??:0
@ 0x4230f0 __tsan_thread_start_func at ??:?
W20260929 15:35:41.224962 1247 tablet.cc:2369] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:35:41.234756 1105 tablet.cc:2369] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7f411b8de6db start_thread at ??:0
@ 0x7f4116d6461f clone at ??:0
I20260929 15:35:41.330754 1482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:41.331195 1482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:41.333000 1482 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
W20260929 15:35:41.337656 739 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099): Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
W20260929 15:35:41.414629 1395 tablet.cc:2369] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:35:41.578027 738 connection.cc:535] client connection to 127.0.100.62:41769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 15:35:41.581434 738 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:35:41.581851 738 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:41.582660 1482 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 15:35:41.697120 1486 raft_consensus.cc:482] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:41.697695 1486 raft_consensus.cc:504] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:41.700170 1486 leader_election.cc:288] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865), 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:41.715433 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4ee8b9bf243339757aa47f51c0203" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f323e4c5c0f4d63bd2de77984299c86" is_pre_election: true
I20260929 15:35:41.716467 1050 raft_consensus.cc:2454] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 0.
I20260929 15:35:41.717969 1131 leader_election.cc:302] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d101b58a89c4d418feb4c9a65cbd21c, 8f323e4c5c0f4d63bd2de77984299c86; no voters:
I20260929 15:35:41.718029 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4ee8b9bf243339757aa47f51c0203" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e5d3d83abe4b5681aeac687d0ae209" is_pre_election: true
I20260929 15:35:41.718848 1486 raft_consensus.cc:2781] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:35:41.718979 1345 raft_consensus.cc:2454] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 0.
I20260929 15:35:41.719184 1486 raft_consensus.cc:482] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:35:41.719513 1486 raft_consensus.cc:3037] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:41.724521 1486 raft_consensus.cc:504] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:41.726186 1486 leader_election.cc:288] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 election: Requested vote from peers 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865), 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:41.727119 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4ee8b9bf243339757aa47f51c0203" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f323e4c5c0f4d63bd2de77984299c86"
I20260929 15:35:41.727305 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4ee8b9bf243339757aa47f51c0203" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e5d3d83abe4b5681aeac687d0ae209"
I20260929 15:35:41.727593 1050 raft_consensus.cc:3037] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:41.728524 1345 raft_consensus.cc:3037] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:41.734026 1050 raft_consensus.cc:2454] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 1.
I20260929 15:35:41.735445 1131 leader_election.cc:302] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d101b58a89c4d418feb4c9a65cbd21c, 8f323e4c5c0f4d63bd2de77984299c86; no voters:
I20260929 15:35:41.736079 1345 raft_consensus.cc:2454] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 1.
I20260929 15:35:41.736605 1486 raft_consensus.cc:2781] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:35:41.738893 1486 raft_consensus.cc:686] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 LEADER]: Becoming Leader. State: Replica: 3d101b58a89c4d418feb4c9a65cbd21c, State: Running, Role: LEADER
I20260929 15:35:41.742597 1486 consensus_queue.cc:237] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
W20260929 15:35:40.121547 1403 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.100.60:37099)
W20260929 15:35:41.582417 1402 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.100.62:41769)
W20260929 15:35:41.239177 405 debug-util.cc:398] Leaking SignalData structure 0x7b080002a0a0 after lost signal to thread 1403
W20260929 15:35:41.876838 405 kernel_stack_watchdog.cc:198] Thread 1403 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/outbound_call.cc:308 for 123ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260929 15:35:41.876150 1402 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.0.100.62:41769, user_credentials={real_user=slave}} blocked reactor thread for 299911us
W20260929 15:35:41.877898 1403 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.0.100.60:37099, user_credentials={real_user=slave}} blocked reactor thread for 1.76317e+06us
W20260929 15:35:41.878400 1402 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.100.62:41769)
I20260929 15:35:41.878540 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.60:37099
--webserver_interface=127.0.100.60
--webserver_port=36831
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260929 15:35:41.892118 1402 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.100.62:41769)
I20260929 15:35:42.164436 1497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:42.164921 1497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:42.166641 1497 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
W20260929 15:35:42.172279 739 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099): Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
W20260929 15:35:42.173673 738 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:35:42.174074 738 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:42.174860 1497 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 15:35:42.260594 1494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:42.261387 1494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:42.261881 1494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:35:42.299037 1494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:35:42.299454 1494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:42.299692 1494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:35:42.299994 1494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:35:42.340138 1494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.60:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.100.60
--webserver_port=36831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:42.341647 1494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:42.343459 1494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:35:42.364269 1509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:42.783708 1517 consensus_queue.cc:1035] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:35:42.815017 1524 consensus_queue.cc:1035] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 15:35:43.103253 1529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:43.103708 1529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:43.105345 1529 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
W20260929 15:35:43.110971 738 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:43.112679 739 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099): Network error: Client connection negotiation failed: client connection to 127.0.100.60:37099: connect: Connection refused (error 111)
I20260929 15:35:43.113359 739 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4211c7490eeb482dafc0d5e1dcbd5506; no voters: 02435ab47dcc45cd9b1b9b37bb1dc178, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:43.114355 1529 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 15:35:42.366492 1494 server_base.cc:1034] running on GCE node
W20260929 15:35:42.364353 1510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:42.364362 1512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:43.668761 1494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:43.671708 1494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:43.673178 1494 hybrid_clock.cc:648] HybridClock initialized: now 1790696143673157 us; error 60 us; skew 500 ppm
I20260929 15:35:43.674144 1494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:43.681005 1494 webserver.cc:460] Webserver started at http://127.0.100.60:36831/ using document root <none> and password file <none>
I20260929 15:35:43.682605 1494 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:43.682992 1494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:43.693724 1494 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:35:43.703500 1542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:43.705123 1494 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260929 15:35:43.705607 1494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
format_stamp: "Formatted at 2026-09-29 15:35:28 on dist-test-slave-0wmn"
I20260929 15:35:43.708599 1494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:43.762678 1494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:43.764251 1494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:43.764758 1494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:43.846668 1494 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.60:37099
I20260929 15:35:43.846837 1595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.60:37099 every 8 connection(s)
I20260929 15:35:43.849660 1494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
I20260929 15:35:43.860167 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 1494
I20260929 15:35:43.860638 1596 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:35:43.869777 1596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Bootstrap starting.
I20260929 15:35:43.887356 1596 log.cc:824] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:43.909334 1596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260929 15:35:43.910219 1596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Bootstrap complete.
I20260929 15:35:43.930500 1596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:43.933384 1596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02435ab47dcc45cd9b1b9b37bb1dc178, State: Initialized, Role: FOLLOWER
I20260929 15:35:43.934289 1596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:43.936986 1600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:43.937716 1600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:43.938079 1596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:35:43.976368 1615 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:43.976744 1615 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: loading cluster ID for follower catalog manager: success
I20260929 15:35:43.985045 1615 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: acquiring CA information for follower catalog manager: success
I20260929 15:35:43.991343 1615 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:35:43.972633 1494 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:35:44.000224 1494 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:35:44.142688 1240 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:44.143252 1240 heartbeater.cc:503] Master 127.0.100.60:37099 was elected leader, sending a full tablet report...
I20260929 15:35:44.151614 1559 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" instance_seqno: 1790696136806861) as {username='slave'} at 127.0.100.2:39735; Asking this server to re-register.
I20260929 15:35:44.153569 1240 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:44.157423 1559 ts_manager.cc:194] Registered new tserver with Master: 3d101b58a89c4d418feb4c9a65cbd21c (127.0.100.2:46845)
I20260929 15:35:44.324046 1390 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:44.327255 1559 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" instance_seqno: 1790696139503740) as {username='slave'} at 127.0.100.3:33467; Asking this server to re-register.
I20260929 15:35:44.327553 1600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:44.328063 1600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:44.329368 1390 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:44.332091 1600 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
I20260929 15:35:44.332588 1559 ts_manager.cc:194] Registered new tserver with Master: 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
W20260929 15:35:44.332437 1545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:44.340302 1545 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:35:44.353979 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" is_pre_election: true
I20260929 15:35:44.354806 761 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02435ab47dcc45cd9b1b9b37bb1dc178 in current term 5: Already voted for candidate db89c29300224c4698b6cd8be3dd3cbc in this term.
I20260929 15:35:44.357421 1546 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:44.358394 1600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 4 FOLLOWER]: Advancing to term 5
I20260929 15:35:44.366670 1600 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 15:35:44.369234 1096 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:44.369848 1096 heartbeater.cc:503] Master 127.0.100.60:37099 was elected leader, sending a full tablet report...
I20260929 15:35:44.373682 1558 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" instance_seqno: 1790696134258512) as {username='slave'} at 127.0.100.1:58521; Asking this server to re-register.
I20260929 15:35:44.376327 1096 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:44.379671 1559 ts_manager.cc:194] Registered new tserver with Master: 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865)
I20260929 15:35:44.450258 1626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:44.450940 1626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:44.452986 1626 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
W20260929 15:35:44.460175 738 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:35:44.468160 1569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" is_pre_election: true
I20260929 15:35:44.469074 1569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4211c7490eeb482dafc0d5e1dcbd5506 in term 5.
I20260929 15:35:44.470363 739 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506; no voters: db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:44.471210 1626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 15:35:44.471554 1626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Starting leader election (detected failure of leader db89c29300224c4698b6cd8be3dd3cbc)
I20260929 15:35:44.471956 1626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 5 FOLLOWER]: Advancing to term 6
I20260929 15:35:44.478554 1626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:44.480302 1626 leader_election.cc:288] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 election: Requested vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 02435ab47dcc45cd9b1b9b37bb1dc178 (127.0.100.60:37099)
I20260929 15:35:44.481504 1569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
I20260929 15:35:44.482142 1569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 5 FOLLOWER]: Advancing to term 6
W20260929 15:35:44.483498 738 leader_election.cc:334] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:35:44.489208 1569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4211c7490eeb482dafc0d5e1dcbd5506 in term 6.
I20260929 15:35:44.490357 739 leader_election.cc:302] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178, 4211c7490eeb482dafc0d5e1dcbd5506; no voters: db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:44.491164 1626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 15:35:44.492460 1626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [term 6 LEADER]: Becoming Leader. State: Replica: 4211c7490eeb482dafc0d5e1dcbd5506, State: Running, Role: LEADER
I20260929 15:35:44.493352 1626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:44.497380 1630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4211c7490eeb482dafc0d5e1dcbd5506. Latest consensus state: current_term: 6 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:44.498154 1630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:44.500895 1632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:35:44.508948 1632 catalog_manager.cc:669] Loaded metadata for table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]
I20260929 15:35:44.513492 1632 tablet_loader.cc:96] loaded metadata for tablet 6ca4ee8b9bf243339757aa47f51c0203 (table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0])
I20260929 15:35:44.514509 1632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:35:44.518538 1632 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:44.518792 1632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:35:44.523414 1632 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:35:44.526391 1632 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Loaded TSK: 0
I20260929 15:35:44.527448 1632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:35:44.530217 1632 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260929 15:35:44.533152 1632 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260929 15:35:44.593591 751 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57584:
table { table_name: "default.Table_95792767565c4269a5550bf589d8a379" } new_table_name: "default.Table_879ae5d00f944b438033d879db00571e" modify_external_catalogs: true
I20260929 15:35:44.594925 752 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57584:
name: "default.Table_fcbf632cc0894e83aa7c8dedc9ff1e58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:35:44.595880 750 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57584:
name: "default.Table_7235c1c5685042129526757568b55d04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:35:44.596453 748 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57584:
name: "default.Table_e96a2bf8603447e6a2d4b4558b4ac762"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:35:44.597608 749 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57584:
name: "default.Table_660e1e2557514985886f76f5f9fe0a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:35:44.599305 746 master_service.cc:946] ReplaceTablet: received request to replace tablet 6ca4ee8b9bf243339757aa47f51c0203 from {username='slave'} at 127.0.0.1:57584
I20260929 15:35:44.603075 747 master_service.cc:946] ReplaceTablet: received request to replace tablet 6ca4ee8b9bf243339757aa47f51c0203 from {username='slave'} at 127.0.0.1:57584
F20260929 15:35:44.607405 917 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_e96a2bf8603447e6a2d4b4558b4ac762
*** Check failure stack trace: ***
I20260929 15:35:44.622248 1569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Refusing update from remote peer 4211c7490eeb482dafc0d5e1dcbd5506: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 10. (index mismatch)
I20260929 15:35:44.623843 1626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 15:35:44.624523 1626 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [LEADER]: Peer 02435ab47dcc45cd9b1b9b37bb1dc178 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260929 15:35:44.624987 738 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 -> Peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Couldn't send request to peer db89c29300224c4698b6cd8be3dd3cbc. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:35:44.628960 1569 pending_rounds.cc:76] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Aborting all ops after (but not including) 5
I20260929 15:35:44.629395 1569 pending_rounds.cc:97] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260929 15:35:44.629827 1569 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7fdf1bd949a0 google::LogMessage::Flush() at ??:0
@ 0x7fdf1bd9470f google::LogMessage::~LogMessage() at ??:0
I20260929 15:35:44.673226 1633 mvcc.cc:205] Tried to move back new op lower bound from 7334691408349032448 to 7334691407851884544. Current Snapshot: MvccSnapshot[applied={T|T < 7334691408349032448}]
I20260929 15:35:44.674288 1600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4211c7490eeb482dafc0d5e1dcbd5506. Latest consensus state: current_term: 6 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:44.679059 1630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:44.680032 1630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506 [sys.catalog]: This master's current role is: LEADER
I20260929 15:35:44.679652 1637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:44.681352 1637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:44.682133 1600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:44.686712 746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Sending DeleteTablet for 3 replicas of tablet 6ca4ee8b9bf243339757aa47f51c0203
@ 0x7fdf26e2faea kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260929 15:35:44.690289 1634 mvcc.cc:205] Tried to move back new op lower bound from 7334691392795643904 to 7334691389369356288. Current Snapshot: MvccSnapshot[applied={T|T < 7334691392795643904 or (T in {7334691392795643904})}]
I20260929 15:35:44.700431 1637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:44.701360 1637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:44.703030 746 catalog_manager.cc:6565] ReplaceTablet: tablet 6ca4ee8b9bf243339757aa47f51c0203 deleted and replaced by tablet 020394adce0a4cbc8b58e8db11b1f16d
I20260929 15:35:44.723661 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca4ee8b9bf243339757aa47f51c0203 with delete_type TABLET_DATA_DELETED (replaced by tablet 020394adce0a4cbc8b58e8db11b1f16d) from {username='slave'} at 127.0.0.1:35566
@ 0x7fdf26e2f33c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260929 15:35:44.737914 1647 tablet_replica.cc:330] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: stopping tablet replica
I20260929 15:35:44.739106 1647 raft_consensus.cc:2229] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:35:44.739838 1647 raft_consensus.cc:2258] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:35:44.741493 1175 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca4ee8b9bf243339757aa47f51c0203 with delete_type TABLET_DATA_DELETED (replaced by tablet 020394adce0a4cbc8b58e8db11b1f16d) from {username='slave'} at 127.0.0.1:48222
I20260929 15:35:44.743077 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 020394adce0a4cbc8b58e8db11b1f16d (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:44.744407 1030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020394adce0a4cbc8b58e8db11b1f16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:44.747575 1174 tablet_service.cc:1467] Processing CreateTablet for tablet 020394adce0a4cbc8b58e8db11b1f16d (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:44.748693 1174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020394adce0a4cbc8b58e8db11b1f16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:44.759243 1647 ts_tablet_manager.cc:1905] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:35:44.766295 1649 tablet_replica.cc:330] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: stopping tablet replica
I20260929 15:35:44.770695 1649 raft_consensus.cc:2229] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:35:44.771849 1649 raft_consensus.cc:2258] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:35:44.780754 1649 ts_tablet_manager.cc:1905] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:35:44.787336 1324 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca4ee8b9bf243339757aa47f51c0203 with delete_type TABLET_DATA_DELETED (replaced by tablet 020394adce0a4cbc8b58e8db11b1f16d) from {username='slave'} at 127.0.0.1:57992
I20260929 15:35:44.788365 1323 tablet_service.cc:1467] Processing CreateTablet for tablet 020394adce0a4cbc8b58e8db11b1f16d (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:44.790083 1323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020394adce0a4cbc8b58e8db11b1f16d. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fdf26e32fd2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260929 15:35:44.807457 1651 tablet_replica.cc:330] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: stopping tablet replica
I20260929 15:35:44.808549 1651 raft_consensus.cc:2229] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:35:44.809208 1651 raft_consensus.cc:2258] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:35:44.811936 1651 ts_tablet_manager.cc:1905] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:35:44.814265 1652 tablet_bootstrap.cc:492] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: Bootstrap starting.
I20260929 15:35:44.819653 1650 tablet_bootstrap.cc:492] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: Bootstrap starting.
I20260929 15:35:44.825233 1647 ts_tablet_manager.cc:1918] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 15:35:44.834693 1652 tablet_bootstrap.cc:654] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: Neither blocks nor log segments found. Creating new log.
@ 0x7fdf26e32f8a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260929 15:35:44.844913 1649 ts_tablet_manager.cc:1918] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 15:35:44.845386 1649 log.cc:1196] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Deleting WAL directory at /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal/wals/6ca4ee8b9bf243339757aa47f51c0203
I20260929 15:35:44.846231 1649 ts_tablet_manager.cc:1939] T 6ca4ee8b9bf243339757aa47f51c0203 P 3d101b58a89c4d418feb4c9a65cbd21c: Deleting consensus metadata
I20260929 15:35:44.852963 739 catalog_manager.cc:4841] TS 3d101b58a89c4d418feb4c9a65cbd21c (127.0.100.2:46845): tablet 6ca4ee8b9bf243339757aa47f51c0203 (table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]) successfully deleted
I20260929 15:35:44.854513 1657 tablet_bootstrap.cc:492] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: Bootstrap starting.
I20260929 15:35:44.863231 1657 tablet_bootstrap.cc:654] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:44.831857 1647 log.cc:1196] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Deleting WAL directory at /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal/wals/6ca4ee8b9bf243339757aa47f51c0203
I20260929 15:35:44.866907 1647 ts_tablet_manager.cc:1939] T 6ca4ee8b9bf243339757aa47f51c0203 P 8f323e4c5c0f4d63bd2de77984299c86: Deleting consensus metadata
I20260929 15:35:44.868939 1650 tablet_bootstrap.cc:654] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:44.873912 739 catalog_manager.cc:4841] TS 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865): tablet 6ca4ee8b9bf243339757aa47f51c0203 (table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]) successfully deleted
I20260929 15:35:44.877699 1652 tablet_bootstrap.cc:492] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: No bootstrap required, opened a new log
I20260929 15:35:44.878222 1652 ts_tablet_manager.cc:1397] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.010s
I20260929 15:35:44.880861 1652 raft_consensus.cc:348] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:44.881661 1652 raft_consensus.cc:374] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:44.881999 1652 raft_consensus.cc:729] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f323e4c5c0f4d63bd2de77984299c86, State: Initialized, Role: FOLLOWER
I20260929 15:35:44.882791 1652 consensus_queue.cc:260] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:44.886029 1657 tablet_bootstrap.cc:492] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: No bootstrap required, opened a new log
I20260929 15:35:44.886596 1657 ts_tablet_manager.cc:1397] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.000s
I20260929 15:35:44.886664 1097 heartbeater.cc:503] Master 127.0.100.61:43161 was elected leader, sending a full tablet report...
I20260929 15:35:44.889309 1657 raft_consensus.cc:348] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:44.898790 746 master_service.cc:946] ReplaceTablet: received request to replace tablet 020394adce0a4cbc8b58e8db11b1f16d from {username='slave'} at 127.0.0.1:57584
@ 0x7fdf26e32f1a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260929 15:35:44.902225 1650 tablet_bootstrap.cc:492] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: No bootstrap required, opened a new log
I20260929 15:35:44.890087 1657 raft_consensus.cc:374] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:44.902779 1650 ts_tablet_manager.cc:1397] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.002s
I20260929 15:35:44.902846 1657 raft_consensus.cc:729] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e5d3d83abe4b5681aeac687d0ae209, State: Initialized, Role: FOLLOWER
I20260929 15:35:44.903963 1657 consensus_queue.cc:260] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:44.904785 1651 ts_tablet_manager.cc:1918] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 15:35:44.905938 1650 raft_consensus.cc:348] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:44.906765 1650 raft_consensus.cc:374] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:44.907047 1650 raft_consensus.cc:729] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d101b58a89c4d418feb4c9a65cbd21c, State: Initialized, Role: FOLLOWER
I20260929 15:35:44.907907 1650 consensus_queue.cc:260] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:44.911016 1241 heartbeater.cc:503] Master 127.0.100.61:43161 was elected leader, sending a full tablet report...
I20260929 15:35:44.913579 747 master_service.cc:946] ReplaceTablet: received request to replace tablet 020394adce0a4cbc8b58e8db11b1f16d from {username='slave'} at 127.0.0.1:57584
I20260929 15:35:44.914592 1652 ts_tablet_manager.cc:1428] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: Time spent starting tablet: real 0.036s user 0.002s sys 0.004s
I20260929 15:35:44.923516 746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4211c7490eeb482dafc0d5e1dcbd5506: Sending DeleteTablet for 3 replicas of tablet 020394adce0a4cbc8b58e8db11b1f16d
I20260929 15:35:44.924624 1650 ts_tablet_manager.cc:1428] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: Time spent starting tablet: real 0.022s user 0.003s sys 0.003s
I20260929 15:35:44.925455 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 020394adce0a4cbc8b58e8db11b1f16d with delete_type TABLET_DATA_DELETED (replaced by tablet d2f63154685b4e62953713199cdac51f) from {username='slave'} at 127.0.0.1:35566
I20260929 15:35:44.926229 1647 tablet_replica.cc:330] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: stopping tablet replica
I20260929 15:35:44.926923 1647 raft_consensus.cc:2229] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 15:35:44.927338 746 catalog_manager.cc:6565] ReplaceTablet: tablet 020394adce0a4cbc8b58e8db11b1f16d deleted and replaced by tablet d2f63154685b4e62953713199cdac51f
I20260929 15:35:44.927402 1647 raft_consensus.cc:2258] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 15:35:44.927933 1174 tablet_service.cc:1514] Processing DeleteTablet for tablet 020394adce0a4cbc8b58e8db11b1f16d with delete_type TABLET_DATA_DELETED (replaced by tablet d2f63154685b4e62953713199cdac51f) from {username='slave'} at 127.0.0.1:48222
I20260929 15:35:44.928661 1649 tablet_replica.cc:330] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: stopping tablet replica
I20260929 15:35:44.929318 1649 raft_consensus.cc:2229] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 15:35:44.929764 1649 raft_consensus.cc:2258] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 15:35:44.932030 1649 ts_tablet_manager.cc:1905] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:35:44.936739 1647 ts_tablet_manager.cc:1905] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:35:44.938235 1651 log.cc:1196] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Deleting WAL directory at /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal/wals/6ca4ee8b9bf243339757aa47f51c0203
I20260929 15:35:44.938409 1323 tablet_service.cc:1514] Processing DeleteTablet for tablet 020394adce0a4cbc8b58e8db11b1f16d with delete_type TABLET_DATA_DELETED (replaced by tablet d2f63154685b4e62953713199cdac51f) from {username='slave'} at 127.0.0.1:57992
I20260929 15:35:44.940660 1651 ts_tablet_manager.cc:1939] T 6ca4ee8b9bf243339757aa47f51c0203 P 20e5d3d83abe4b5681aeac687d0ae209: Deleting consensus metadata
I20260929 15:35:44.944079 739 catalog_manager.cc:4841] TS 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189): tablet 6ca4ee8b9bf243339757aa47f51c0203 (table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]) successfully deleted
W20260929 15:35:44.957278 739 catalog_manager.cc:4821] TS 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189): delete failed for tablet 020394adce0a4cbc8b58e8db11b1f16d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 020394adce0a4cbc8b58e8db11b1f16d already in progress: creating tablet
I20260929 15:35:44.959149 1657 ts_tablet_manager.cc:1428] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: Time spent starting tablet: real 0.072s user 0.003s sys 0.007s
I20260929 15:35:44.975904 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d2f63154685b4e62953713199cdac51f (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:44.977021 1030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f63154685b4e62953713199cdac51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:44.979023 1391 heartbeater.cc:503] Master 127.0.100.61:43161 was elected leader, sending a full tablet report...
I20260929 15:35:44.982192 1649 ts_tablet_manager.cc:1918] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 15:35:44.982709 1649 log.cc:1196] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: Deleting WAL directory at /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-1/wal/wals/020394adce0a4cbc8b58e8db11b1f16d
I20260929 15:35:44.983484 1649 ts_tablet_manager.cc:1939] T 020394adce0a4cbc8b58e8db11b1f16d P 3d101b58a89c4d418feb4c9a65cbd21c: Deleting consensus metadata
I20260929 15:35:44.985800 1323 tablet_service.cc:1514] Processing DeleteTablet for tablet 020394adce0a4cbc8b58e8db11b1f16d with delete_type TABLET_DATA_DELETED (replaced by tablet d2f63154685b4e62953713199cdac51f) from {username='slave'} at 127.0.0.1:57992
I20260929 15:35:44.987850 1651 tablet_replica.cc:330] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: stopping tablet replica
I20260929 15:35:44.988549 1651 raft_consensus.cc:2229] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 15:35:44.989154 1651 raft_consensus.cc:2258] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 15:35:44.989121 1174 tablet_service.cc:1467] Processing CreateTablet for tablet d2f63154685b4e62953713199cdac51f (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:44.990257 1174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f63154685b4e62953713199cdac51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:44.991791 1651 ts_tablet_manager.cc:1905] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:35:45.002086 1323 tablet_service.cc:1467] Processing CreateTablet for tablet d2f63154685b4e62953713199cdac51f (DEFAULT_TABLE table=default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:35:45.003444 1323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f63154685b4e62953713199cdac51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:35:45.004392 739 catalog_manager.cc:4841] TS 3d101b58a89c4d418feb4c9a65cbd21c (127.0.100.2:46845): tablet 020394adce0a4cbc8b58e8db11b1f16d (table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]) successfully deleted
I20260929 15:35:45.011680 1652 tablet_bootstrap.cc:492] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86: Bootstrap starting.
I20260929 15:35:45.017499 1652 tablet_bootstrap.cc:654] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:45.019806 1647 ts_tablet_manager.cc:1918] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 15:35:45.021490 1647 log.cc:1196] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: Deleting WAL directory at /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-0/wal/wals/020394adce0a4cbc8b58e8db11b1f16d
I20260929 15:35:45.022372 1647 ts_tablet_manager.cc:1939] T 020394adce0a4cbc8b58e8db11b1f16d P 8f323e4c5c0f4d63bd2de77984299c86: Deleting consensus metadata
@ 0x7fdf26e32ee2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260929 15:35:45.039568 739 catalog_manager.cc:4841] TS 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865): tablet 020394adce0a4cbc8b58e8db11b1f16d (table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]) successfully deleted
I20260929 15:35:45.042068 1657 tablet_bootstrap.cc:492] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209: Bootstrap starting.
I20260929 15:35:45.047312 1650 tablet_bootstrap.cc:492] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c: Bootstrap starting.
I20260929 15:35:45.055415 1650 tablet_bootstrap.cc:654] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:45.057380 1657 tablet_bootstrap.cc:654] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209: Neither blocks nor log segments found. Creating new log.
I20260929 15:35:45.064114 1651 ts_tablet_manager.cc:1918] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 15:35:45.064509 1651 log.cc:1196] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: Deleting WAL directory at /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/ts-2/wal/wals/020394adce0a4cbc8b58e8db11b1f16d
I20260929 15:35:45.064625 1650 tablet_bootstrap.cc:492] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c: No bootstrap required, opened a new log
I20260929 15:35:45.065169 1651 ts_tablet_manager.cc:1939] T 020394adce0a4cbc8b58e8db11b1f16d P 20e5d3d83abe4b5681aeac687d0ae209: Deleting consensus metadata
I20260929 15:35:45.065236 1650 ts_tablet_manager.cc:1397] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20260929 15:35:45.067718 739 catalog_manager.cc:4841] TS 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189): tablet 020394adce0a4cbc8b58e8db11b1f16d (table default.table_95792767565c4269a5550bf589d8a379 [id=566074959d5b48c4b94b6df736b3eed0]) successfully deleted
I20260929 15:35:45.069476 1652 tablet_bootstrap.cc:492] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86: No bootstrap required, opened a new log
I20260929 15:35:45.069945 1652 ts_tablet_manager.cc:1397] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.016s
I20260929 15:35:45.069320 1650 raft_consensus.cc:348] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:45.070549 1650 raft_consensus.cc:374] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:45.070916 1650 raft_consensus.cc:729] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d101b58a89c4d418feb4c9a65cbd21c, State: Initialized, Role: FOLLOWER
I20260929 15:35:45.071542 1657 tablet_bootstrap.cc:492] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209: No bootstrap required, opened a new log
I20260929 15:35:45.071959 1657 ts_tablet_manager.cc:1397] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.002s
I20260929 15:35:45.071887 1650 consensus_queue.cc:260] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:45.072602 1652 raft_consensus.cc:348] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:45.074661 1657 raft_consensus.cc:348] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:45.075335 1657 raft_consensus.cc:374] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:45.075587 1657 raft_consensus.cc:729] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e5d3d83abe4b5681aeac687d0ae209, State: Initialized, Role: FOLLOWER
I20260929 15:35:45.073354 1652 raft_consensus.cc:374] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:35:45.076560 1657 consensus_queue.cc:260] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:45.077296 1652 raft_consensus.cc:729] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f323e4c5c0f4d63bd2de77984299c86, State: Initialized, Role: FOLLOWER
I20260929 15:35:45.077977 1652 consensus_queue.cc:260] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:45.080024 1652 ts_tablet_manager.cc:1428] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86: Time spent starting tablet: real 0.010s user 0.001s sys 0.003s
I20260929 15:35:45.082623 1657 ts_tablet_manager.cc:1428] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
@ 0x7fdf26e321de std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5805ba std::__1::function<>::operator()() at ??:?
I20260929 15:35:45.109586 1650 ts_tablet_manager.cc:1428] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c: Time spent starting tablet: real 0.044s user 0.004s sys 0.003s
@ 0x7fdf1df45dc7 kudu::Thread::SuperviseThread() at ??:0
@ 0x4230f0 __tsan_thread_start_func at ??:?
I20260929 15:35:45.130661 745 master_service.cc:946] ReplaceTablet: received request to replace tablet d2f63154685b4e62953713199cdac51f from {username='slave'} at 127.0.0.1:57584
I20260929 15:35:45.140751 746 master_service.cc:946] ReplaceTablet: received request to replace tablet d2f63154685b4e62953713199cdac51f from {username='slave'} at 127.0.0.1:57584
@ 0x7fdf1f5b36db start_thread at ??:0
I20260929 15:35:45.159880 1625 raft_consensus.cc:482] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:45.160447 1625 raft_consensus.cc:504] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
@ 0x7fdf1aa3961f clone at ??:0
I20260929 15:35:45.164147 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f63154685b4e62953713199cdac51f" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f323e4c5c0f4d63bd2de77984299c86" is_pre_election: true
I20260929 15:35:45.164250 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f63154685b4e62953713199cdac51f" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e5d3d83abe4b5681aeac687d0ae209" is_pre_election: true
I20260929 15:35:45.164853 1345 raft_consensus.cc:2454] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 0.
I20260929 15:35:45.164896 1050 raft_consensus.cc:2454] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 0.
I20260929 15:35:45.162631 1625 leader_election.cc:288] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865), 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:45.166160 1131 leader_election.cc:302] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e5d3d83abe4b5681aeac687d0ae209, 3d101b58a89c4d418feb4c9a65cbd21c; no voters:
I20260929 15:35:45.167541 1625 raft_consensus.cc:2781] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:35:45.167799 1625 raft_consensus.cc:482] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:35:45.167996 1625 raft_consensus.cc:3037] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:45.175125 1625 raft_consensus.cc:504] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
I20260929 15:35:45.177911 1625 leader_election.cc:288] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 election: Requested vote from peers 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865), 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:45.178648 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f63154685b4e62953713199cdac51f" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e5d3d83abe4b5681aeac687d0ae209"
I20260929 15:35:45.179191 1345 raft_consensus.cc:3037] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:45.184280 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f63154685b4e62953713199cdac51f" candidate_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f323e4c5c0f4d63bd2de77984299c86"
I20260929 15:35:45.184894 1050 raft_consensus.cc:3037] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:35:45.185197 1345 raft_consensus.cc:2454] T d2f63154685b4e62953713199cdac51f P 20e5d3d83abe4b5681aeac687d0ae209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 1.
I20260929 15:35:45.188135 1131 leader_election.cc:302] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e5d3d83abe4b5681aeac687d0ae209, 3d101b58a89c4d418feb4c9a65cbd21c; no voters:
I20260929 15:35:45.189057 1625 raft_consensus.cc:2781] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:35:45.191287 1050 raft_consensus.cc:2454] T d2f63154685b4e62953713199cdac51f P 8f323e4c5c0f4d63bd2de77984299c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d101b58a89c4d418feb4c9a65cbd21c in term 1.
I20260929 15:35:45.193416 1625 raft_consensus.cc:686] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [term 1 LEADER]: Becoming Leader. State: Replica: 3d101b58a89c4d418feb4c9a65cbd21c, State: Running, Role: LEADER
I20260929 15:35:45.195212 1625 consensus_queue.cc:237] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 } } peers { permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46845 } } peers { permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 } }
W20260929 15:35:45.377944 1403 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.100.61:43161)
W20260929 15:35:45.379238 1403 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.100.61:43161)
W20260929 15:35:45.380708 1403 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.100.61:43161)
W20260929 15:35:45.381978 1131 connection.cc:535] client connection to 127.0.100.61:43161 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 15:35:45.656124 1679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:45.656594 1679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:45.658301 1679 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:45.667645 1545 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:45.668311 1546 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:45.668700 1546 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:45.669610 1679 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:46.021843 1684 consensus_queue.cc:1035] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:35:46.045477 1684 consensus_queue.cc:1035] T d2f63154685b4e62953713199cdac51f P 3d101b58a89c4d418feb4c9a65cbd21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 15:35:46.341555 1696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4211c7490eeb482dafc0d5e1dcbd5506)
I20260929 15:35:46.342064 1696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:46.343710 1696 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:46.348407 1545 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:46.348937 1546 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:46.349359 1546 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:46.350015 1696 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:46.492584 400 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu with pid 1494
I20260929 15:35:47.256098 400 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
/tmp/dist-test-taskYukZNI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.60:37099
--webserver_interface=127.0.100.60
--webserver_port=36831
--builtin_ntp_servers=127.0.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260929 15:35:47.611532 1709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:35:47.612171 1709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:35:47.612660 1709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:35:47.646194 1709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:35:47.646553 1709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:35:47.646838 1709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:35:47.647099 1709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:35:47.684397 1709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:38279
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.100.60:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.100.60
--webserver_port=36831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:50:29 UTC on ccycloud.quasar-iyffyt.root.comops.site
build id 4680449
TSAN enabled
I20260929 15:35:47.686084 1709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:35:47.688091 1709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:35:47.706826 1719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:35:48.852706 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf420 after lost signal to thread 1252
W20260929 15:35:48.853940 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000d7ce0 after lost signal to thread 1389
W20260929 15:35:47.706837 1720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:47.709832 1709 server_base.cc:1034] running on GCE node
W20260929 15:35:47.708592 1722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:35:49.038200 1709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:35:49.041841 1709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:35:49.043430 1709 hybrid_clock.cc:648] HybridClock initialized: now 1790696149043334 us; error 110 us; skew 500 ppm
I20260929 15:35:49.044375 1709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:35:49.051924 1709 webserver.cc:460] Webserver started at http://127.0.100.60:36831/ using document root <none> and password file <none>
I20260929 15:35:49.052964 1709 fs_manager.cc:359] Metadata directory not provided
I20260929 15:35:49.053227 1709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:35:49.062390 1709 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 15:35:49.067655 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:35:49.069022 1709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260929 15:35:49.069465 1709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data,/tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
uuid: "02435ab47dcc45cd9b1b9b37bb1dc178"
format_stamp: "Formatted at 2026-09-29 15:35:28 on dist-test-slave-0wmn"
I20260929 15:35:49.072151 1709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:35:49.171528 1709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:35:49.173812 1709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:35:49.174458 1709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:35:49.272544 1709 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.60:37099
I20260929 15:35:49.272804 1797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.60:37099 every 8 connection(s)
I20260929 15:35:49.275731 1709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYukZNI/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790696104978279-400-0/minicluster-data/master-2/data/info.pb
I20260929 15:35:49.279875 400 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYukZNI/build/tsan/bin/kudu as pid 1709
I20260929 15:35:49.303936 1798 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:35:49.318531 1798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Bootstrap starting.
I20260929 15:35:49.340963 1798 log.cc:824] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Log is configured to *not* fsync() on all Append() calls
I20260929 15:35:49.381892 1798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=1 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:35:49.383087 1798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: Bootstrap complete.
I20260929 15:35:49.405324 1798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:49.407919 1798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02435ab47dcc45cd9b1b9b37bb1dc178, State: Initialized, Role: FOLLOWER
I20260929 15:35:49.408910 1798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 6.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:49.412102 1803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } } }
I20260929 15:35:49.413530 1803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:35:49.414129 1798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:35:49.448952 1818 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:35:49.456265 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:35:49.489634 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:35:49.606930 1096 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:49.617050 1762 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8f323e4c5c0f4d63bd2de77984299c86" instance_seqno: 1790696134258512) as {username='slave'} at 127.0.100.1:36089; Asking this server to re-register.
I20260929 15:35:49.619228 1096 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:49.623751 1762 ts_manager.cc:194] Registered new tserver with Master: 8f323e4c5c0f4d63bd2de77984299c86 (127.0.100.1:45865)
I20260929 15:35:49.624193 1390 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:49.627866 1762 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "20e5d3d83abe4b5681aeac687d0ae209" instance_seqno: 1790696139503740) as {username='slave'} at 127.0.100.3:36385; Asking this server to re-register.
I20260929 15:35:49.629370 1390 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:49.631354 1240 heartbeater.cc:347] Connected to a master server at 127.0.100.60:37099
I20260929 15:35:49.633324 1762 ts_manager.cc:194] Registered new tserver with Master: 20e5d3d83abe4b5681aeac687d0ae209 (127.0.100.3:36189)
I20260929 15:35:49.635142 1763 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3d101b58a89c4d418feb4c9a65cbd21c" instance_seqno: 1790696136806861) as {username='slave'} at 127.0.100.2:46591; Asking this server to re-register.
I20260929 15:35:49.636320 1240 heartbeater.cc:464] Registering TS with master...
I20260929 15:35:49.639407 1763 ts_manager.cc:194] Registered new tserver with Master: 3d101b58a89c4d418feb4c9a65cbd21c (127.0.100.2:46845)
I20260929 15:35:49.912595 1803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:49.913084 1803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:49.915710 1803 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:49.917073 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:49.923679 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:49.924860 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:49.925354 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:49.926286 1803 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:50.459923 1818 catalog_manager.cc:1259] Loaded cluster ID: a00aa9ebffc14307a1257e481d3f650f
I20260929 15:35:50.460279 1818 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: loading cluster ID for follower catalog manager: success
I20260929 15:35:50.470767 1818 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: acquiring CA information for follower catalog manager: success
I20260929 15:35:50.478705 1818 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:35:50.610072 1827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:50.610862 1827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:50.628971 1827 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:50.639985 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:35:50.640658 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:35:50.641235 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:50.642690 1827 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:51.227012 1831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:51.227512 1831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:51.229213 1831 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:51.235169 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:51.235670 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:51.236019 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:51.236605 1831 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:51.730589 400 master-stress-test.cc:578] Killing test thread 1/10
I20260929 15:35:52.631949 1837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:52.632364 1837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:52.633966 1837 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:52.639292 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:52.639806 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:52.640142 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:52.640719 1837 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:54.492957 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:35:54.493463 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:35:54.592324 1846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:54.592895 1846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:54.594614 1846 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:54.599323 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:35:54.599987 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:35:54.600406 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:54.601125 1846 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:57.143723 1863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:35:57.144316 1863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:35:57.146337 1863 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:35:57.148667 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260929 15:35:57.151835 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:35:57.152357 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:35:57.152734 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:35:57.153458 1863 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:35:59.496857 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:35:59.497475 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:02.222844 1898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:02.223412 1898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:02.225489 1898 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:02.228767 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:36:02.232157 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:36:02.232810 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:36:02.233403 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:02.234426 1898 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:36:04.500378 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:04.500738 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:05.953217 1923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:05.954437 1923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:05.959261 1923 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:05.965289 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:36:05.965842 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:36:05.966182 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:05.966811 1923 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:36:09.504730 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:09.505319 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:09.679883 1947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:09.680569 1947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:09.683068 1947 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:09.684242 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260929 15:36:09.687884 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:36:09.688452 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:36:09.688791 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:09.689590 1947 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:36:14.508728 1977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:14.509485 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:14.509308 1977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:14.509927 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:14.511489 1977 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:14.515512 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:36:14.516001 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:36:14.516352 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:14.516943 1977 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:36:19.513465 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:19.513967 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:24.517166 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:24.517751 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:24.915055 2042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:24.915808 2042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:24.919350 2042 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:24.921403 1750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260929 15:36:24.925468 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:36:24.926007 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:36:24.926467 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:24.927109 2042 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260929 15:36:25.503204 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2b40 after lost signal to thread 1252
W20260929 15:36:25.504321 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000435a0 after lost signal to thread 1389
I20260929 15:36:29.575562 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:29.576159 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:36:30.188390 1794 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0040 after lost signal to thread 1713
W20260929 15:36:30.189177 1794 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3520 after lost signal to thread 1797
I20260929 15:36:33.298072 2081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:33.298552 2081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:33.300513 2081 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:33.303012 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:36:33.306836 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:36:33.307449 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:36:33.307883 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:33.308609 2081 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:36:34.579448 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:34.579938 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:37.413862 2094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:37.414376 2094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:37.417174 2094 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:37.425453 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:36:37.425948 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:36:37.426317 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:37.427049 2094 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:36:39.583696 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:39.584201 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:36:40.344201 1392 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (123 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:36:40.834589 1097 heartbeater.cc:644] Failed to heartbeat to 127.0.100.61:43161 (111 consecutive failures): Network error: Failed to ping master at 127.0.100.61:43161: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:36:40.947754 1242 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (125 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:36:44.360188 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc4c0 after lost signal to thread 1108
W20260929 15:36:44.361205 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000d35a0 after lost signal to thread 1239
I20260929 15:36:44.587002 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:44.587502 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:49.590811 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:49.591266 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:54.594571 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:54.594962 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:36:54.691731 2128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:36:54.692399 2128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:36:54.694841 2128 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:36:54.696260 1750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260929 15:36:54.699784 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:36:54.700410 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:36:54.700809 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:36:54.701516 2128 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:36:59.598608 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:36:59.599175 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:04.603305 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:04.603928 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:08.310402 2156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:37:08.310948 2156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:37:08.312603 2156 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:37:08.314801 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:37:08.317876 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:08.318434 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:37:08.318732 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:37:08.319341 2156 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:37:09.607409 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:09.607929 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:14.610371 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:14.610821 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:14.762308 2171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:37:14.762775 2171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:37:14.764442 2171 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:37:14.765810 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:37:14.768786 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:14.769327 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:37:14.769687 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:37:14.770226 2171 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:37:19.615023 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:19.615528 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:23.405344 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:37:23.406068 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:37:23.408743 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:37:23.412070 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:37:23.415920 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:23.416783 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:37:23.417084 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:37:23.417858 2189 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:37:24.618355 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:24.618762 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:24.886401 2194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:37:24.886955 2194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:37:24.889156 2194 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:37:24.895058 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:24.895503 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:37:24.895853 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:37:24.896418 2194 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:37:29.622764 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:29.623270 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:37:33.070729 1092 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9cc0 after lost signal to thread 945
W20260929 15:37:33.071782 1092 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4ee0 after lost signal to thread 1095
I20260929 15:37:34.626034 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:34.626537 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:37:36.776003 1794 debug-util.cc:398] Leaking SignalData structure 0x7b08000780e0 after lost signal to thread 1713
W20260929 15:37:36.776763 1794 debug-util.cc:398] Leaking SignalData structure 0x7b08000afa20 after lost signal to thread 1797
I20260929 15:37:39.669090 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:39.669816 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:37:40.617470 1392 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (243 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:41.120085 1097 heartbeater.cc:644] Failed to heartbeat to 127.0.100.61:43161 (231 consecutive failures): Network error: Failed to ping master at 127.0.100.61:43161: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:37:41.229907 1242 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (245 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:37:44.672632 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:44.672986 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:44.839684 2233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:37:44.841012 2233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
W20260929 15:37:44.847503 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20260929 15:37:44.850493 2233 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:37:44.860817 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:44.861544 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:37:44.861893 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:37:44.862620 2233 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:37:49.553942 1752 maintenance_manager.cc:419] P 02435ab47dcc45cd9b1b9b37bb1dc178: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033373
I20260929 15:37:49.665827 1751 maintenance_manager.cc:643] P 02435ab47dcc45cd9b1b9b37bb1dc178: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.111s user 0.049s sys 0.029s Metrics: {"bytes_written":7418,"cfile_init":1,"dirs.queue_time_us":1005,"dirs.run_cpu_time_us":409,"dirs.run_wall_time_us":1213,"drs_written":1,"lbm_read_time_us":370,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3196,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":8,"thread_start_us":1084,"threads_started":2,"wal-append.queue_time_us":943}
I20260929 15:37:49.669124 1752 maintenance_manager.cc:419] P 02435ab47dcc45cd9b1b9b37bb1dc178: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 573 bytes on disk
I20260929 15:37:49.671419 1751 maintenance_manager.cc:643] P 02435ab47dcc45cd9b1b9b37bb1dc178: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":370,"lbm_reads_lt_1ms":4}
I20260929 15:37:49.677585 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:49.678117 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:37:54.061007 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000b60a0 after lost signal to thread 1108
W20260929 15:37:54.061939 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb7e0 after lost signal to thread 1239
I20260929 15:37:54.681360 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:54.681749 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:37:54.783811 2269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:37:54.784303 2269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:37:54.786005 2269 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:37:54.787169 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:37:54.790927 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:54.791560 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:37:54.791879 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:37:54.792474 2269 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:37:56.360668 2282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:37:56.361106 2282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:37:56.362673 2282 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:37:56.366654 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:37:56.367018 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:37:56.367309 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:37:56.367800 2282 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260929 15:37:57.925612 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000d81c0 after lost signal to thread 1252
W20260929 15:37:57.926955 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000426c0 after lost signal to thread 1389
I20260929 15:37:59.685046 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:37:59.685534 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:03.395373 2323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:38:03.395918 2323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:38:03.397717 2323 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:38:03.400036 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260929 15:38:03.403342 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:38:03.403927 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:38:03.404218 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:38:03.404838 2323 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:38:04.688449 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:04.688817 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:09.692672 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:09.693204 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:14.696584 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:14.697018 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:38:16.067212 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000dbc80 after lost signal to thread 1252
W20260929 15:38:16.068197 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1a40 after lost signal to thread 1389
I20260929 15:38:19.701090 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:19.701653 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:21.545576 2459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:38:21.546185 2459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:38:21.548496 2459 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:38:21.551230 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:38:21.554433 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:38:21.555049 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:38:21.555402 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:38:21.556030 2459 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:38:24.705117 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:24.705578 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:29.709560 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:29.710196 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:38:31.454926 1386 debug-util.cc:398] Leaking SignalData structure 0x7b0800040280 after lost signal to thread 1252
W20260929 15:38:31.455955 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1c00 after lost signal to thread 1389
I20260929 15:38:34.163286 2591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:38:34.163866 2591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:38:34.165606 2591 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:38:34.167917 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:38:34.170989 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:38:34.171404 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:38:34.171718 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:38:34.172257 2591 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:38:34.712958 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:34.713408 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:39.717453 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:39.717931 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:38:40.901607 1392 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (363 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:38:41.431007 1097 heartbeater.cc:644] Failed to heartbeat to 127.0.100.61:43161 (351 consecutive failures): Network error: Failed to ping master at 127.0.100.61:43161: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:38:41.575765 1242 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (365 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:38:44.721827 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:44.722309 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:38:48.052735 1092 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0d20 after lost signal to thread 945
W20260929 15:38:48.053735 1092 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0600 after lost signal to thread 1095
I20260929 15:38:49.726219 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:49.726781 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:53.484668 2841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:38:53.485164 2841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:38:53.487499 2841 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:38:53.489600 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:38:53.492466 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:38:53.492929 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:38:53.493237 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:38:53.493852 2841 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:38:54.730299 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:54.730788 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:38:55.335500 2870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:38:55.336103 2870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:38:55.337890 2870 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:38:55.342749 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:38:55.343216 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:38:55.343593 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:38:55.344214 2870 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260929 15:38:56.252643 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1aa0 after lost signal to thread 1252
W20260929 15:38:56.253732 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6820 after lost signal to thread 1389
I20260929 15:38:56.726168 2893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:38:56.726940 2893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:38:56.730391 2893 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:38:56.747169 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:38:56.748039 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:38:56.748471 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:38:56.749080 2893 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:38:59.734299 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:38:59.734788 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:39:04.738826 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:04.739216 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:39:09.742556 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:09.742975 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:39:12.381506 3098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:39:12.382081 3098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:39:12.384341 3098 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:39:12.386348 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260929 15:39:12.388993 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:39:12.389541 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:39:12.389885 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:39:12.390483 3098 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:39:14.746169 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:14.746582 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:39:17.979480 1092 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0f20 after lost signal to thread 945
W20260929 15:39:17.980428 1092 debug-util.cc:398] Leaking SignalData structure 0x7b08000429c0 after lost signal to thread 1095
I20260929 15:39:19.750159 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:19.750620 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:39:24.097517 1794 debug-util.cc:398] Leaking SignalData structure 0x7b080009f060 after lost signal to thread 1713
W20260929 15:39:24.098104 1794 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae1e0 after lost signal to thread 1797
W20260929 15:39:24.615386 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1f40 after lost signal to thread 1108
W20260929 15:39:24.616693 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7100 after lost signal to thread 1239
I20260929 15:39:24.774999 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:24.775606 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:39:29.779274 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:29.779752 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:39:30.654485 3333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:39:30.655009 3333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:39:30.657610 3333 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:39:30.659745 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:39:30.662817 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:39:30.663314 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:39:30.663682 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:39:30.664279 3333 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:39:34.782563 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:34.783001 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:39:39.786881 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:39.787468 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:39:41.185328 1392 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (483 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:39:41.471621 1097 heartbeater.cc:644] Failed to heartbeat to 127.0.100.61:43161 (470 consecutive failures): Network error: Failed to ping master at 127.0.100.61:43161: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:39:41.988785 1242 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (485 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:39:42.475234 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5380 after lost signal to thread 1108
W20260929 15:39:42.476465 1236 debug-util.cc:398] Leaking SignalData structure 0x7b0800043260 after lost signal to thread 1239
I20260929 15:39:43.583632 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:39:43.584257 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:39:43.586540 3508 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:39:43.589105 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:39:43.592281 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:39:43.592934 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:39:43.593343 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:39:43.594038 3508 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:39:44.790755 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:44.791251 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:39:47.327296 1794 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2720 after lost signal to thread 1713
W20260929 15:39:47.327870 1794 debug-util.cc:398] Leaking SignalData structure 0x7b0800052c80 after lost signal to thread 1797
W20260929 15:39:49.590688 1794 debug-util.cc:398] Leaking SignalData structure 0x7b0800052120 after lost signal to thread 1713
W20260929 15:39:49.591332 1794 debug-util.cc:398] Leaking SignalData structure 0x7b0800051980 after lost signal to thread 1797
I20260929 15:39:49.858418 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:49.858871 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:39:51.286841 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000d7600 after lost signal to thread 1252
W20260929 15:39:51.287748 1386 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5b00 after lost signal to thread 1389
I20260929 15:39:54.862049 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:54.862449 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:39:56.777482 3701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:39:56.777994 3701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:39:56.779804 3701 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:39:56.782158 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:39:56.785288 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:39:56.786010 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:39:56.786350 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:39:56.787016 3701 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:39:59.865976 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:39:59.866472 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:04.869781 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:04.870186 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:07.398017 3855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:40:07.398546 3855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:40:07.400506 3855 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:40:07.402681 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:40:07.405403 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:40:07.405830 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:40:07.406119 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:40:07.406656 3855 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:40:09.873587 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:09.874034 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:14.877177 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:14.877684 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:19.881511 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:19.882102 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:24.884763 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:24.885198 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:25.181437 4133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:40:25.181993 4133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:40:25.184137 4133 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:40:25.185328 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260929 15:40:25.188174 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:40:25.188730 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:40:25.189004 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:40:25.189704 4133 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:40:28.239799 4185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:40:28.240646 4185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:40:28.243178 4185 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:40:28.248435 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:40:28.248950 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:40:28.249313 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:40:28.249946 4185 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:40:29.888175 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:29.888587 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:29.925940 4210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:40:29.926319 4210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:40:29.928341 4210 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:40:29.933179 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:40:29.933676 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:40:29.934067 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:40:29.934680 4210 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260929 15:40:32.907338 1794 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae580 after lost signal to thread 1713
W20260929 15:40:32.908001 1794 debug-util.cc:398] Leaking SignalData structure 0x7b080009bc20 after lost signal to thread 1797
I20260929 15:40:34.907111 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:34.907577 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:40:37.810280 1236 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd8e0 after lost signal to thread 1108
W20260929 15:40:37.811240 1236 debug-util.cc:398] Leaking SignalData structure 0x7b0800043c00 after lost signal to thread 1239
I20260929 15:40:39.911033 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:39.911459 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:40:41.634378 1392 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (603 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:40:41.883597 1099 heartbeater.cc:644] Failed to heartbeat to 127.0.100.62:41769 (602 consecutive failures): Network error: Failed to ping master at 127.0.100.62:41769: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
W20260929 15:40:42.382691 1241 heartbeater.cc:644] Failed to heartbeat to 127.0.100.61:43161 (592 consecutive failures): Network error: Failed to ping master at 127.0.100.61:43161: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
I20260929 15:40:42.684101 4369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:40:42.684623 4369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89c29300224c4698b6cd8be3dd3cbc" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 41769 } } peers { permanent_uuid: "4211c7490eeb482dafc0d5e1dcbd5506" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 43161 } } peers { permanent_uuid: "02435ab47dcc45cd9b1b9b37bb1dc178" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 37099 } }
I20260929 15:40:42.686884 4369 leader_election.cc:288] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769), 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161)
W20260929 15:40:42.689079 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260929 15:40:42.692659 1750 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4211c7490eeb482dafc0d5e1dcbd5506 (127.0.100.61:43161): Network error: Client connection negotiation failed: client connection to 127.0.100.61:43161: connect: Connection refused (error 111)
W20260929 15:40:42.693487 1749 leader_election.cc:334] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer db89c29300224c4698b6cd8be3dd3cbc (127.0.100.62:41769): Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111)
I20260929 15:40:42.693753 1749 leader_election.cc:302] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02435ab47dcc45cd9b1b9b37bb1dc178; no voters: 4211c7490eeb482dafc0d5e1dcbd5506, db89c29300224c4698b6cd8be3dd3cbc
I20260929 15:40:42.694283 4369 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02435ab47dcc45cd9b1b9b37bb1dc178 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 15:40:44.915203 1709 master_runner.cc:186] Error getting master registration for 127.0.100.62:41769: Network error: Client connection negotiation failed: client connection to 127.0.100.62:41769: connect: Connection refused (error 111),
I20260929 15:40:44.915650 1709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:40:45.151345 1752 maintenance_manager.cc:419] P 02435ab47dcc45cd9b1b9b37bb1dc178: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 573 bytes on disk
I20260929 15:40:45.165628 1751 maintenance_manager.cc:643] P 02435ab47dcc45cd9b1b9b37bb1dc178: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"dirs.queue_time_us":1081,"dirs.run_cpu_time_us":358,"dirs.run_wall_time_us":359,"thread_start_us":2863,"threads_started":2,"wal-append.queue_time_us":2521}
W20260929 15:40:45.520977 1403 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 1)
W20260929 15:40:45.521708 1403 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 1)
W20260929 15:40:45.522152 1403 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 1)
F20260929 15:40:45.535012 1413 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e96a2bf8603447e6a2d4b4558b4ac762 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 1)
*** Check failure stack trace: ***
F20260929 15:40:45.536749 1417 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fcbf632cc0894e83aa7c8dedc9ff1e58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.100.62:41769,127.0.100.61:43161,127.0.100.60:37099, num_attempts: 1)
*** Aborted at 1790696445 (unix time) try "date -d @1790696445" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000190) received by PID 400 (TID 0x7f87c4cfe700) from PID 400; stack trace: ***
@ 0x2b0af0 __tsan::CallUserSignalHandler() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b31e0 rtl_sigaction() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
*** Check failure stack trace: ***
@ 0x7f87d4ff6980 (unknown) at ??:0
@ 0x7f87d02b3e87 gsignal at ??:0
@ 0x7f87d02b57f1 abort at ??:0
@ 0x2b4f37 __interceptor_abort at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f87d32e7df4 google::LogMessage::SendToLog() at ??:0
@ 0x7f87d32e89a0 google::LogMessage::Flush() at ??:0
@ 0x7f87d32edadb google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x3572c4 kudu::MasterStressTest::CreateTableThread() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x3874ce _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
@ 0x387381 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x386acb std::__1::__thread_proxy<>() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2b11b0 __tsan_thread_start_func at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f87d4feb6db start_thread at ??:0
@ 0x7f87d039661f clone at ??:0