Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-08-09T01:45:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:45:16Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskFVJal6/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskFVJal6/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/security.properties
2026-08-09 01:45:26: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskFVJal6/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskFVJal6/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260809 01:45:38.794896 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.62:33531
--webserver_interface=127.27.115.62
--webserver_port=0
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260809 01:45:39.192003 28426 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:45:39.192762 28426 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:45:39.193282 28426 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 01:45:39.229598 28426 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:45:39.230039 28426 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:45:39.230347 28426 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:45:39.230647 28426 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:45:39.276669 28426 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.62:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.115.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:45:39.279139 28426 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:45:39.281635 28426 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 01:45:39.301107 28432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:39.301282 28433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:40.718967 28435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:40.721823 28434 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1416 milliseconds
I20260809 01:45:40.722167 28426 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260809 01:45:40.723822 28426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:45:40.727779 28426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:45:40.731654 28426 hybrid_clock.cc:648] HybridClock initialized: now 1786239940731534 us; error 159 us; skew 500 ppm
I20260809 01:45:40.733043 28426 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:45:40.742344 28426 webserver.cc:533] Webserver started at http://127.27.115.62:35605/ using document root <none> and password file <none>
I20260809 01:45:40.744280 28426 fs_manager.cc:362] Metadata directory not provided
I20260809 01:45:40.744632 28426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:45:40.745503 28426 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:45:40.751372 28426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/instance:
uuid: "9b0e02ad573e476983ee8c2d623d2501"
format_stamp: "Formatted at 2026-08-09 01:45:40 on dist-test-slave-dhph"
I20260809 01:45:40.753021 28426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal/instance:
uuid: "9b0e02ad573e476983ee8c2d623d2501"
format_stamp: "Formatted at 2026-08-09 01:45:40 on dist-test-slave-dhph"
I20260809 01:45:40.763717 28426 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.006s sys 0.005s
I20260809 01:45:40.773466 28442 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:40.775621 28426 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260809 01:45:40.776297 28426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
uuid: "9b0e02ad573e476983ee8c2d623d2501"
format_stamp: "Formatted at 2026-08-09 01:45:40 on dist-test-slave-dhph"
I20260809 01:45:40.776736 28426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:45:40.846117 28426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:45:40.848470 28426 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:45:40.849175 28426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:45:40.969632 28426 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.62:33531
I20260809 01:45:40.969811 28497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.62:33531 every 8 connection(s)
I20260809 01:45:40.973608 28426 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/info.pb
I20260809 01:45:40.983851 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 28426
I20260809 01:45:40.984462 28108 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal/instance
I20260809 01:45:40.988082 28498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:45:40.988881 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.61:37213
--webserver_interface=127.27.115.61
--webserver_port=0
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260809 01:45:41.011806 28498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:45:41.061201 28498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
W20260809 01:45:41.064224 28445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
W20260809 01:45:41.069607 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:45:41.119338 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 1
W20260809 01:45:41.125814 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:45:41.194401 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 2
W20260809 01:45:41.201090 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:45:41.292644 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 3
W20260809 01:45:41.301970 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:45:41.446113 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 4
W20260809 01:45:41.453487 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
W20260809 01:45:41.463948 28499 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:45:41.464695 28499 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:45:41.465193 28499 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 01:45:41.502871 28499 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:45:41.503311 28499 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:45:41.503646 28499 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:45:41.503949 28499 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:45:41.549067 28499 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.61:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.115.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:45:41.550809 28499 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:45:41.552819 28499 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 01:45:41.570178 28509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:41.753224 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 5
W20260809 01:45:41.766001 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:45:42.313721 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 6
W20260809 01:45:42.347491 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:45:43.410681 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 7
W20260809 01:45:43.446903 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
W20260809 01:45:42.982300 28508 debug-util.cc:398] Leaking SignalData structure 0x7b08000374a0 after lost signal to thread 28499
W20260809 01:45:41.574865 28510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:43.501066 28499 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.926s user 0.832s sys 1.080s
W20260809 01:45:43.502470 28499 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.928s user 0.832s sys 1.081s
W20260809 01:45:43.505952 28513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:43.507296 28512 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1926 milliseconds
I20260809 01:45:43.507414 28499 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260809 01:45:43.509709 28499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:45:43.514146 28499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:45:43.515856 28499 hybrid_clock.cc:648] HybridClock initialized: now 1786239943515675 us; error 175 us; skew 500 ppm
I20260809 01:45:43.517014 28499 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:45:43.527839 28499 webserver.cc:533] Webserver started at http://127.27.115.61:42297/ using document root <none> and password file <none>
I20260809 01:45:43.529670 28499 fs_manager.cc:362] Metadata directory not provided
I20260809 01:45:43.530035 28499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:45:43.530777 28499 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:45:43.537598 28499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/instance:
uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
format_stamp: "Formatted at 2026-08-09 01:45:43 on dist-test-slave-dhph"
I20260809 01:45:43.539788 28499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal/instance:
uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
format_stamp: "Formatted at 2026-08-09 01:45:43 on dist-test-slave-dhph"
I20260809 01:45:43.550463 28499 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.005s
I20260809 01:45:43.561753 28524 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:43.563642 28499 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260809 01:45:43.564121 28499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
format_stamp: "Formatted at 2026-08-09 01:45:43 on dist-test-slave-dhph"
I20260809 01:45:43.564565 28499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:45:43.629058 28499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:45:43.631052 28499 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:45:43.631685 28499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:45:43.768137 28499 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.61:37213
I20260809 01:45:43.768363 28583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.61:37213 every 8 connection(s)
I20260809 01:45:43.771591 28499 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/info.pb
I20260809 01:45:43.773511 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 28499
I20260809 01:45:43.774593 28108 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal/instance
I20260809 01:45:43.779516 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.60:45299
--webserver_interface=127.27.115.60
--webserver_port=0
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260809 01:45:43.783452 28584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:45:43.799944 28584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:45:43.829456 28584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:45:43.849759 28584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } has no permanent_uuid. Determining permanent_uuid...
W20260809 01:45:43.854135 28526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:45:43.863973 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:43.913767 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 1
W20260809 01:45:43.920722 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:43.989239 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 2
W20260809 01:45:43.994549 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:44.086107 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 3
W20260809 01:45:44.092859 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:44.236470 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 4
W20260809 01:45:44.242735 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:45:44.248145 28585 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:45:44.249136 28585 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:45:44.249657 28585 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 01:45:44.288457 28585 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:45:44.288990 28585 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:45:44.289307 28585 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:45:44.289603 28585 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:45:44.331435 28585 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.60:45299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.115.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:45:44.332949 28585 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:45:44.335395 28585 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 01:45:44.351055 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:44.542234 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 5
W20260809 01:45:44.553128 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:45.101020 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 6
W20260809 01:45:45.129317 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:45.537820 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } attempt: 8
I20260809 01:45:45.592264 28498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } has no permanent_uuid. Determining permanent_uuid...
W20260809 01:45:45.620296 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:45.686017 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 1
W20260809 01:45:45.695842 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:45.750746 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 2
W20260809 01:45:45.761837 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:45.838532 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 3
W20260809 01:45:45.848630 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:46.004407 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 4
W20260809 01:45:46.014966 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:45:45.758602 28594 debug-util.cc:398] Leaking SignalData structure 0x7b08000374a0 after lost signal to thread 28585
W20260809 01:45:44.355476 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:46.091991 28594 kernel_stack_watchdog.cc:198] Thread 28585 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260809 01:45:46.093765 28585 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.739s user 0.570s sys 1.159s
W20260809 01:45:46.094833 28585 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.741s user 0.570s sys 1.160s
W20260809 01:45:46.096966 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:46.098985 28597 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1741 milliseconds
I20260809 01:45:46.099148 28585 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260809 01:45:46.100879 28585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:45:46.104416 28585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:45:46.106009 28585 hybrid_clock.cc:648] HybridClock initialized: now 1786239946105974 us; error 62 us; skew 500 ppm
I20260809 01:45:46.107153 28585 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:45:46.116628 28585 webserver.cc:533] Webserver started at http://127.27.115.60:36659/ using document root <none> and password file <none>
I20260809 01:45:46.117798 28585 fs_manager.cc:362] Metadata directory not provided
I20260809 01:45:46.118089 28585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:45:46.118865 28585 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:45:46.124167 28585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data/instance:
uuid: "118ef761cca1412eac9223a88e991ff0"
format_stamp: "Formatted at 2026-08-09 01:45:46 on dist-test-slave-dhph"
I20260809 01:45:46.125574 28585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/wal/instance:
uuid: "118ef761cca1412eac9223a88e991ff0"
format_stamp: "Formatted at 2026-08-09 01:45:46 on dist-test-slave-dhph"
I20260809 01:45:46.138075 28585 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.006s
I20260809 01:45:46.147233 28613 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:46.149803 28585 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.007s sys 0.001s
I20260809 01:45:46.150816 28585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/wal
uuid: "118ef761cca1412eac9223a88e991ff0"
format_stamp: "Formatted at 2026-08-09 01:45:46 on dist-test-slave-dhph"
I20260809 01:45:46.151391 28585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:45:46.190021 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 7
W20260809 01:45:46.199703 28584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:46.220744 28585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:45:46.224172 28585 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:45:46.224788 28585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:45:46.311682 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 5
W20260809 01:45:46.316809 28443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20260809 01:45:46.322695 28498 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:45:46.373661 28585 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.60:45299
I20260809 01:45:46.373867 28673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.60:45299 every 8 connection(s)
I20260809 01:45:46.377542 28585 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/data/info.pb
I20260809 01:45:46.385869 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 28585
I20260809 01:45:46.386366 28108 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-2/wal/instance
I20260809 01:45:46.387218 28675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:45:46.405745 28675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:45:46.453673 28675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:45:46.475548 28675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:45:46.505972 28675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: Bootstrap starting.
I20260809 01:45:46.520098 28675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: Neither blocks nor log segments found. Creating new log.
I20260809 01:45:46.523504 28675 log.cc:826] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: Log is configured to *not* fsync() on all Append() calls
I20260809 01:45:46.530573 28675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: No bootstrap required, opened a new log
I20260809 01:45:46.564335 28675 raft_consensus.cc:359] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:46.565222 28675 raft_consensus.cc:385] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:45:46.565471 28675 raft_consensus.cc:740] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118ef761cca1412eac9223a88e991ff0, State: Initialized, Role: FOLLOWER
I20260809 01:45:46.566936 28675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:46.573107 28688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:46.577279 28688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:45:46.580451 28675 sys_catalog.cc:565] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 01:45:46.640060 28704 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:45:46.640695 28704 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:45:46.632964 28585 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: OK, instance_id { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" instance_seqno: 1786239940930835 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 01:45:46.666882 28585 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:45:46.733621 28622 tablet.cc:2406] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:45:46.844417 28498 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 6
I20260809 01:45:46.854312 28688 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:45:46.855526 28688 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:46.875002 28688 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
I20260809 01:45:46.884409 28498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Bootstrap starting.
I20260809 01:45:46.896361 28498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Neither blocks nor log segments found. Creating new log.
I20260809 01:45:46.901132 28498 log.cc:826] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Log is configured to *not* fsync() on all Append() calls
I20260809 01:45:46.914013 28498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: No bootstrap required, opened a new log
I20260809 01:45:46.945240 28498 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:46.947342 28498 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:45:46.947899 28498 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0e02ad573e476983ee8c2d623d2501, State: Initialized, Role: FOLLOWER
I20260809 01:45:46.949808 28498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:46.955775 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:46.957962 28498 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:45:46.957741 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:45:46.958280 28472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:46.959710 28472 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 118ef761cca1412eac9223a88e991ff0 in term 0.
I20260809 01:45:46.966013 28619 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501; no voters:
I20260809 01:45:46.968803 28688 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:45:46.969305 28688 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:45:46.969615 28688 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:45:46.978912 28688 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:46.984937 28472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501"
I20260809 01:45:46.985751 28472 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:45:46.997377 28472 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 118ef761cca1412eac9223a88e991ff0 in term 1.
I20260809 01:45:47.002075 28688 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 1 election: Requested vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
I20260809 01:45:47.009692 28619 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501; no voters:
I20260809 01:45:47.011279 28688 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:45:47.033381 28688 raft_consensus.cc:697] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 1 LEADER]: Becoming Leader. State: Replica: 118ef761cca1412eac9223a88e991ff0, State: Running, Role: LEADER
I20260809 01:45:47.037935 28688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:47.103394 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118ef761cca1412eac9223a88e991ff0. Latest consensus state: current_term: 1 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:47.105863 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:45:47.110734 28724 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
W20260809 01:45:47.120357 28449 tablet.cc:2406] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:45:47.123741 28724 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260809 01:45:47.176855 28472 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Refusing update from remote peer 118ef761cca1412eac9223a88e991ff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:45:47.179334 28715 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 01:45:47.193714 28732 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:45:47.199393 28732 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:45:47.168610 28426 master_runner.cc:187] Error getting master registration for 127.27.115.61:37213: OK, instance_id { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" instance_seqno: 1786239943726804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 01:45:47.251387 28426 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:45:47.277593 28730 mvcc.cc:204] Tried to move back new op lower bound from 7316438823545606144 to 7316438823134609408. Current Snapshot: MvccSnapshot[applied={T|T < 7316438823545606144}]
I20260809 01:45:47.281236 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118ef761cca1412eac9223a88e991ff0. Latest consensus state: current_term: 1 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:47.282452 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:45:47.290407 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:47.291536 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:45:47.293239 28724 catalog_manager.cc:1383] Generated new cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:45:47.293601 28724 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260809 01:45:47.321235 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:47.322166 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:45:47.426111 28724 catalog_manager.cc:1406] Generated new certificate authority record
I20260809 01:45:47.428726 28724 catalog_manager.cc:1540] Loading token signing keys...
I20260809 01:45:47.477164 28724 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: Generated new TSK 0
I20260809 01:45:47.484083 28724 catalog_manager.cc:1550] Initializing in-progress tserver states...
W20260809 01:45:47.497394 28618 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.115.61:37213 timed out after 0.631s (SENT)
I20260809 01:45:47.528255 28724 catalog_manager.cc:1563] Loading latest processed Hive Metastore notification log event ID...
I20260809 01:45:47.538934 28724 catalog_manager.cc:5941] Last processed Hive Metastore notification event ID: -1
I20260809 01:45:47.567778 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.115.1:0
--local_ip_for_outbound_sockets=127.27.115.1
--webserver_interface=127.27.115.1
--webserver_port=0
--tserver_master_addrs=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20260809 01:45:48.149556 28618 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Timed out: RequestConsensusVote RPC to 127.27.115.61:37213 timed out after 0.631s (SENT)
W20260809 01:45:48.182904 28618 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Timed out: RequestConsensusVote RPC to 127.27.115.61:37213 timed out after 0.589s (SENT)
I20260809 01:45:48.290246 28584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } attempt: 8
I20260809 01:45:48.318010 28732 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:45:48.322280 28732 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: loading cluster ID for follower catalog manager: success
I20260809 01:45:48.364923 28584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Bootstrap starting.
W20260809 01:45:48.383549 28739 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:45:48.384347 28739 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:45:48.385376 28739 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260809 01:45:48.387207 28584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Neither blocks nor log segments found. Creating new log.
I20260809 01:45:48.393324 28584 log.cc:826] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Log is configured to *not* fsync() on all Append() calls
I20260809 01:45:48.407869 28732 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: acquiring CA information for follower catalog manager: success
I20260809 01:45:48.421881 28584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: No bootstrap required, opened a new log
I20260809 01:45:48.456316 28732 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:45:48.496065 28584 raft_consensus.cc:359] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:48.497195 28584 raft_consensus.cc:385] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:45:48.497592 28584 raft_consensus.cc:740] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9cc5c3257b466c8087bf6fc15b7284, State: Initialized, Role: FOLLOWER
I20260809 01:45:48.499164 28584 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:48.510936 28584 sys_catalog.cc:565] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:45:48.511214 28552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:48.512207 28552 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 118ef761cca1412eac9223a88e991ff0 in term 0.
W20260809 01:45:48.513031 28552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36990 (request call id 4) took 993 ms (client timeout 631 ms). Trace:
W20260809 01:45:48.513468 28552 rpcz_store.cc:269] 0809 01:45:47.519395 (+ 0us) service_pool.cc:167] Inserting onto call queue
0809 01:45:47.520819 (+ 1424us) service_pool.cc:224] Handling call
0809 01:45:48.512928 (+992109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260809 01:45:48.514941 28554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
I20260809 01:45:48.515679 28554 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:45:48.518913 28555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
W20260809 01:45:48.520156 28555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36990 (request call id 1) took 1656 ms (client timeout 631 ms). Trace:
W20260809 01:45:48.520525 28555 rpcz_store.cc:269] 0809 01:45:46.864094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0809 01:45:46.864279 (+ 185us) service_pool.cc:224] Handling call
0809 01:45:48.520038 (+1655759us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260809 01:45:48.522029 28551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
W20260809 01:45:48.526633 28551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36990 (request call id 5) took 933 ms (client timeout 589 ms). Trace:
W20260809 01:45:48.527319 28551 rpcz_store.cc:269] 0809 01:45:47.593120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0809 01:45:47.593302 (+ 182us) service_pool.cc:224] Handling call
0809 01:45:48.526518 (+933216us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260809 01:45:48.530092 28739 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:45:48.532270 28739 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.115.1
I20260809 01:45:48.553776 28554 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 118ef761cca1412eac9223a88e991ff0 in term 1.
W20260809 01:45:48.554947 28554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36990 (request call id 2) took 1553 ms (client timeout 589 ms). Trace:
W20260809 01:45:48.555414 28554 rpcz_store.cc:269] 0809 01:45:47.001811 (+ 0us) service_pool.cc:167] Inserting onto call queue
0809 01:45:47.002044 (+ 233us) service_pool.cc:224] Handling call
0809 01:45:48.554845 (+1552801us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260809 01:45:48.557542 28553 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Refusing update from remote peer 118ef761cca1412eac9223a88e991ff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:45:48.508976 28747 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:48.560143 28747 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:45:48.562000 28553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36990 (request call id 3) took 1400 ms. Trace:
I20260809 01:45:48.562417 28553 rpcz_store.cc:276] 0809 01:45:47.162518 (+ 0us) service_pool.cc:167] Inserting onto call queue
0809 01:45:47.163647 (+ 1129us) service_pool.cc:224] Handling call
0809 01:45:48.556850 (+1393203us) raft_consensus.cc:1471] Updating replica for 0 ops
0809 01:45:48.558118 (+ 1268us) raft_consensus.cc:1697] Filling consensus response to leader.
0809 01:45:48.561617 (+ 3499us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x7b54000203d0. stack: 00007ff9265cdcf3 00007ff9265cdbf0 00007ff92617b574 0000000000516de4 0000000000516d79 00007ff92d6ce951 00007ff92d764c8d 00007ff92e7c3229 00007ff92ce41d34 00007ff92ce41cc2 00007ff92ce41bf1 00007ff92ce41b6c 00007ff92ce40e22 00007ff92b43007c 00007ff92b42f4b6 00007ff92b42ee5f
0809 01:45:48.561912 (+ 295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80450304}
I20260809 01:45:48.565179 28688 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:45:48.634302 28739 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.115.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.115.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.115.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:45:48.636826 28739 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:45:48.639830 28739 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260809 01:45:48.657526 28747 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118ef761cca1412eac9223a88e991ff0. Latest consensus state: current_term: 1 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:48.658785 28747 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:45:48.661073 28758 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:48.662474 28758 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 01:45:48.679497 28766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:48.702060 28776 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 01:45:49.072190 28758 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:49.073201 28758 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:49.078769 28758 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:49.079099 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:49.080969 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:49.084012 28529 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:49.085220 28758 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:49.185853 28688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:49.186712 28688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:45:49.724591 28776 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:45:49.724944 28776 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: loading cluster ID for follower catalog manager: success
I20260809 01:45:49.743489 28784 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:49.752957 28776 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: acquiring CA information for follower catalog manager: success
I20260809 01:45:49.753962 28784 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:49.766299 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:49.760030 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:49.769285 28784 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:49.770386 28445 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:49.787791 28785 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:49.794054 28776 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:45:50.339910 28786 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:50.350894 28786 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:50.355545 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:50.356305 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:50.357187 28786 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:50.358990 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:50.359871 28786 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:50.467834 28787 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:50.468838 28787 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:50.474709 28787 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:50.474432 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:50.480103 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:50.482862 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:50.489300 28787 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 01:45:50.072474 28764 debug-util.cc:398] Leaking SignalData structure 0x7b0800037500 after lost signal to thread 28739
W20260809 01:45:50.499624 28764 kernel_stack_watchdog.cc:198] Thread 28739 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 393ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260809 01:45:50.499784 28739 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.824s user 0.386s sys 0.610s
W20260809 01:45:50.500192 28739 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.824s user 0.386s sys 0.610s
W20260809 01:45:48.681160 28767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:50.515731 28739 server_base.cc:1061] running on GCE node
W20260809 01:45:50.517841 28791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:50.519680 28739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:45:50.528502 28739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:45:50.532518 28739 hybrid_clock.cc:648] HybridClock initialized: now 1786239950532437 us; error 59 us; skew 500 ppm
I20260809 01:45:50.534133 28739 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:45:50.556202 28739 webserver.cc:533] Webserver started at http://127.27.115.1:39337/ using document root <none> and password file <none>
I20260809 01:45:50.558063 28739 fs_manager.cc:362] Metadata directory not provided
I20260809 01:45:50.558543 28739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:45:50.559373 28739 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:45:50.566877 28739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data/instance:
uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0"
format_stamp: "Formatted at 2026-08-09 01:45:50 on dist-test-slave-dhph"
I20260809 01:45:50.569502 28739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal/instance:
uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0"
format_stamp: "Formatted at 2026-08-09 01:45:50 on dist-test-slave-dhph"
I20260809 01:45:50.589133 28739 fs_manager.cc:696] Time spent creating directory manager: real 0.018s user 0.009s sys 0.006s
I20260809 01:45:50.605674 28796 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:50.607564 28739 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.008s sys 0.002s
I20260809 01:45:50.608153 28739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal
uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0"
format_stamp: "Formatted at 2026-08-09 01:45:50 on dist-test-slave-dhph"
I20260809 01:45:50.608834 28739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:45:50.728967 28739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:45:50.731591 28739 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:45:50.732270 28739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:45:50.759591 28739 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260809 01:45:50.783178 28739 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:45:50.783746 28739 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260809 01:45:50.784160 28739 ts_tablet_manager.cc:616] Registered 0 tablets
I20260809 01:45:50.784439 28739 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:50.974289 28846 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:50.975102 28846 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:50.979439 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:50.981169 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:50.984006 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:50.987087 28846 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:50.988399 28846 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:51.025231 28861 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:51.026144 28861 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:51.032298 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:51.037099 28861 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:51.038376 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:51.040573 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:51.042341 28861 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:51.190927 28739 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.1:40405
I20260809 01:45:51.191147 28928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.1:40405 every 8 connection(s)
I20260809 01:45:51.195214 28739 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/data/info.pb
I20260809 01:45:51.200502 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 28739
I20260809 01:45:51.201354 28108 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal/instance
I20260809 01:45:51.247810 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.115.2:0
--local_ip_for_outbound_sockets=127.27.115.2
--webserver_interface=127.27.115.2
--webserver_port=0
--tserver_master_addrs=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260809 01:45:51.363255 28929 heartbeater.cc:344] Connected to a master server at 127.27.115.60:45299
I20260809 01:45:51.363852 28929 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:51.365526 28929 heartbeater.cc:507] Master 127.27.115.60:45299 requested a full tablet report, sending...
I20260809 01:45:51.370112 28934 heartbeater.cc:344] Connected to a master server at 127.27.115.62:33531
I20260809 01:45:51.370561 28934 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:51.372378 28934 heartbeater.cc:507] Master 127.27.115.62:33531 requested a full tablet report, sending...
I20260809 01:45:51.375530 28930 heartbeater.cc:344] Connected to a master server at 127.27.115.61:37213
I20260809 01:45:51.376196 28930 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:51.378255 28930 heartbeater.cc:507] Master 127.27.115.61:37213 requested a full tablet report, sending...
I20260809 01:45:51.380177 28460 ts_manager.cc:194] Registered new tserver with Master: f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405)
I20260809 01:45:51.380997 28635 ts_manager.cc:194] Registered new tserver with Master: f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405)
I20260809 01:45:51.387187 28544 ts_manager.cc:194] Registered new tserver with Master: f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405)
I20260809 01:45:51.395918 28635 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.115.1:57499
W20260809 01:45:51.767055 28494 debug-util.cc:398] Leaking SignalData structure 0x7b080007afe0 after lost signal to thread 28427
I20260809 01:45:51.901847 28929 heartbeater.cc:499] Master 127.27.115.60:45299 was elected leader, sending a full tablet report...
W20260809 01:45:51.953639 28940 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:45:51.954487 28940 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:45:51.955641 28940 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 01:45:52.024647 28940 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:45:52.026546 28940 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.115.2
I20260809 01:45:52.099143 28940 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.115.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.115.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.115.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:45:52.101181 28940 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:45:52.103905 28940 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 01:45:52.128541 28948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:52.414031 28956 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:52.414616 28956 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:52.417678 28956 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:52.420478 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:52.431252 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:52.459458 28445 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:52.460879 28956 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:52.521178 28957 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:52.522432 28957 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:52.531178 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:52.531414 28957 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:52.536626 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:52.551050 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:52.553455 28957 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:53.076772 28958 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:53.080699 28958 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:53.087467 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:53.093698 28958 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:53.095228 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:53.097297 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:53.098383 28958 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:53.390903 28959 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:53.392266 28959 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:53.395988 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:53.396067 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:53.398445 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:53.407688 28960 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:53.409175 28959 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:45:53.549278 28947 debug-util.cc:398] Leaking SignalData structure 0x7b0800037500 after lost signal to thread 28940
W20260809 01:45:54.063208 28950 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1928 milliseconds
W20260809 01:45:54.062175 28940 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.930s user 0.232s sys 0.640s
W20260809 01:45:54.064775 28940 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.933s user 0.232s sys 0.640s
W20260809 01:45:54.065016 28952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:54.065198 28940 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260809 01:45:52.132566 28949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:54.072870 28940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:45:54.078126 28940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:45:54.080037 28940 hybrid_clock.cc:648] HybridClock initialized: now 1786239954079978 us; error 107 us; skew 500 ppm
I20260809 01:45:54.081676 28940 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:45:54.094805 28940 webserver.cc:533] Webserver started at http://127.27.115.2:35483/ using document root <none> and password file <none>
I20260809 01:45:54.096431 28940 fs_manager.cc:362] Metadata directory not provided
I20260809 01:45:54.096823 28940 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:45:54.098403 28940 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:45:54.107153 28940 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data/instance:
uuid: "1b148a8b196342d7ade139bdb80e5145"
format_stamp: "Formatted at 2026-08-09 01:45:54 on dist-test-slave-dhph"
I20260809 01:45:54.109491 28940 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal/instance:
uuid: "1b148a8b196342d7ade139bdb80e5145"
format_stamp: "Formatted at 2026-08-09 01:45:54 on dist-test-slave-dhph"
I20260809 01:45:54.125331 28940 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.011s sys 0.000s
I20260809 01:45:54.140594 28965 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:54.143451 28940 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260809 01:45:54.144207 28940 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal
uuid: "1b148a8b196342d7ade139bdb80e5145"
format_stamp: "Formatted at 2026-08-09 01:45:54 on dist-test-slave-dhph"
I20260809 01:45:54.144991 28940 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:45:54.287694 28940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:45:54.290076 28940 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:45:54.290802 28940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:45:54.295425 28940 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260809 01:45:54.303552 28940 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:45:54.303937 28940 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:54.304296 28940 ts_tablet_manager.cc:616] Registered 0 tablets
I20260809 01:45:54.304548 28940 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:54.478165 29035 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:54.478931 29035 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:54.483337 29035 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:54.484300 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:54.487321 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:54.490084 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:54.491678 29035 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:54.656544 28940 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.2:44481
I20260809 01:45:54.656692 29093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.2:44481 every 8 connection(s)
I20260809 01:45:54.659539 28940 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/data/info.pb
I20260809 01:45:54.661674 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 28940
I20260809 01:45:54.662236 28108 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal/instance
I20260809 01:45:54.688869 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.115.3:0
--local_ip_for_outbound_sockets=127.27.115.3
--webserver_interface=127.27.115.3
--webserver_port=0
--tserver_master_addrs=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260809 01:45:54.747785 29094 heartbeater.cc:344] Connected to a master server at 127.27.115.60:45299
I20260809 01:45:54.748956 29094 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:54.751269 29094 heartbeater.cc:507] Master 127.27.115.60:45299 requested a full tablet report, sending...
I20260809 01:45:54.761910 28635 ts_manager.cc:194] Registered new tserver with Master: 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2:44481)
I20260809 01:45:54.765420 28635 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.115.2:42569
I20260809 01:45:54.780311 29095 heartbeater.cc:344] Connected to a master server at 127.27.115.61:37213
I20260809 01:45:54.780845 29095 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:54.781920 29095 heartbeater.cc:507] Master 127.27.115.61:37213 requested a full tablet report, sending...
I20260809 01:45:54.787818 28544 ts_manager.cc:194] Registered new tserver with Master: 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2:44481)
I20260809 01:45:54.789359 29100 heartbeater.cc:344] Connected to a master server at 127.27.115.62:33531
I20260809 01:45:54.789937 29100 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:54.792115 29100 heartbeater.cc:507] Master 127.27.115.62:33531 requested a full tablet report, sending...
I20260809 01:45:54.800195 28460 ts_manager.cc:194] Registered new tserver with Master: 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2:44481)
I20260809 01:45:54.968088 29105 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:54.969154 29105 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:54.973168 29105 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:54.973978 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:54.976606 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:54.982728 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:54.983974 29105 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:55.274559 29094 heartbeater.cc:499] Master 127.27.115.60:45299 was elected leader, sending a full tablet report...
W20260809 01:45:55.314956 29101 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:45:55.315661 29101 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:45:55.316393 29101 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 01:45:55.354564 29101 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:45:55.355860 29101 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.115.3
I20260809 01:45:55.419585 29101 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.115.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.115.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.115.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:45:55.421614 29101 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:45:55.424647 29101 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 01:45:55.444558 29113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:45:55.536158 29119 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:55.538300 29119 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:55.547614 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:55.556568 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:55.560141 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:55.575724 29120 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:55.576596 29119 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:55.604096 29121 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:55.605468 29121 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:55.611994 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:55.621300 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:55.625433 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:55.630913 29121 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:55.633183 29122 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:56.566226 29124 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:56.567098 29124 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:56.570184 29124 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:56.571861 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:56.572553 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:56.577091 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:56.579205 29124 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:56.635272 29125 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:56.636137 29125 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:56.641855 29125 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:56.642659 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:56.645120 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:56.659861 28529 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:56.662171 29125 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 01:45:56.853509 29112 debug-util.cc:398] Leaking SignalData structure 0x7b0800037500 after lost signal to thread 29101
W20260809 01:45:57.334771 29112 kernel_stack_watchdog.cc:198] Thread 29101 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 405ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260809 01:45:57.336277 29101 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.892s user 0.580s sys 1.168s
W20260809 01:45:57.336740 29101 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.893s user 0.580s sys 1.168s
W20260809 01:45:55.448504 29114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:57.337828 29116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:45:57.342298 29115 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1891 milliseconds
I20260809 01:45:57.342372 29101 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260809 01:45:57.343915 29101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:45:57.346750 29101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:45:57.348388 29101 hybrid_clock.cc:648] HybridClock initialized: now 1786239957348301 us; error 99 us; skew 500 ppm
I20260809 01:45:57.349601 29101 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:45:57.358685 29101 webserver.cc:533] Webserver started at http://127.27.115.3:40841/ using document root <none> and password file <none>
I20260809 01:45:57.359835 29101 fs_manager.cc:362] Metadata directory not provided
I20260809 01:45:57.360060 29101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:45:57.360689 29101 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:45:57.365778 29101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data/instance:
uuid: "b758ff267d534924b3a3204689550643"
format_stamp: "Formatted at 2026-08-09 01:45:57 on dist-test-slave-dhph"
I20260809 01:45:57.367192 29101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal/instance:
uuid: "b758ff267d534924b3a3204689550643"
format_stamp: "Formatted at 2026-08-09 01:45:57 on dist-test-slave-dhph"
I20260809 01:45:57.381685 29101 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260809 01:45:57.392386 29130 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:57.394375 29101 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.003s
I20260809 01:45:57.395057 29101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal
uuid: "b758ff267d534924b3a3204689550643"
format_stamp: "Formatted at 2026-08-09 01:45:57 on dist-test-slave-dhph"
I20260809 01:45:57.395598 29101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:45:57.466030 29101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:45:57.468253 29101 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:45:57.468851 29101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:45:57.474575 29101 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260809 01:45:57.482336 29101 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:45:57.482642 29101 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:57.482895 29101 ts_tablet_manager.cc:616] Registered 0 tablets
I20260809 01:45:57.483193 29101 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:45:57.541895 29161 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:57.542646 29161 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:57.545717 29161 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:57.547662 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:57.547712 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:57.549780 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:57.550813 29161 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:45:57.623564 29196 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:57.624854 29196 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:57.627694 29196 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:57.628804 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:57.629187 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:57.629766 28471 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9cc5c3257b466c8087bf6fc15b7284 in term 1.
I20260809 01:45:57.631989 28529 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501, ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0
I20260809 01:45:57.633196 29196 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 01:45:57.633806 29196 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:57.634331 29196 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:45:57.648427 29196 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:57.653640 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501"
I20260809 01:45:57.656772 28553 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Rejecting Update request from peer 118ef761cca1412eac9223a88e991ff0 for earlier term 1. Current term is 2. Ops: []
I20260809 01:45:57.657825 29196 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 election: Requested vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:57.658927 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0"
I20260809 01:45:57.661657 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:57.662214 29201 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 01:45:57.663267 29196 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 01:45:57.673547 29212 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 01:45:57.674539 29212 raft_consensus.cc:740] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 118ef761cca1412eac9223a88e991ff0, State: Running, Role: LEADER
I20260809 01:45:57.676532 29212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:57.679057 29212 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:45:57.804574 29101 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.3:40957
I20260809 01:45:57.805101 29263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.3:40957 every 8 connection(s)
I20260809 01:45:57.808709 29101 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/data/info.pb
I20260809 01:45:57.811021 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 29101
I20260809 01:45:57.812355 28108 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal/instance
I20260809 01:45:57.872720 29264 heartbeater.cc:344] Connected to a master server at 127.27.115.60:45299
I20260809 01:45:57.873358 29264 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:57.873939 29265 heartbeater.cc:344] Connected to a master server at 127.27.115.61:37213
I20260809 01:45:57.874359 29265 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:57.876545 29265 heartbeater.cc:507] Master 127.27.115.61:37213 requested a full tablet report, sending...
I20260809 01:45:57.875415 29264 heartbeater.cc:507] Master 127.27.115.60:45299 requested a full tablet report, sending...
I20260809 01:45:57.876793 29271 heartbeater.cc:344] Connected to a master server at 127.27.115.62:33531
I20260809 01:45:57.878808 29271 heartbeater.cc:461] Registering TS with master...
I20260809 01:45:57.881071 28544 ts_manager.cc:194] Registered new tserver with Master: b758ff267d534924b3a3204689550643 (127.27.115.3:40957)
I20260809 01:45:57.882494 29271 heartbeater.cc:507] Master 127.27.115.62:33531 requested a full tablet report, sending...
I20260809 01:45:57.884963 28634 ts_manager.cc:194] Registered new tserver with Master: b758ff267d534924b3a3204689550643 (127.27.115.3:40957)
I20260809 01:45:57.886477 28460 ts_manager.cc:194] Registered new tserver with Master: b758ff267d534924b3a3204689550643 (127.27.115.3:40957)
I20260809 01:45:57.914984 28108 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260809 01:45:58.141149 29282 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:58.142067 29282 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:58.145819 29282 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:58.146907 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:58.148366 28553 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b0e02ad573e476983ee8c2d623d2501 in current term 2: Already voted for candidate ba9cc5c3257b466c8087bf6fc15b7284 in this term.
I20260809 01:45:58.148602 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:58.149807 28647 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0e02ad573e476983ee8c2d623d2501 in term 2.
I20260809 01:45:58.151692 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501; no voters: ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:58.152799 29282 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 01:45:58.153237 29282 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:45:58.153498 29282 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:45:58.164512 29282 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:58.167798 29282 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 election: Requested vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:58.169064 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
I20260809 01:45:58.169440 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0"
I20260809 01:45:58.170025 28553 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9b0e02ad573e476983ee8c2d623d2501 in current term 2: Already voted for candidate ba9cc5c3257b466c8087bf6fc15b7284 in this term.
I20260809 01:45:58.177503 28647 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0e02ad573e476983ee8c2d623d2501 in term 2.
I20260809 01:45:58.179260 28443 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501; no voters: ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:58.180119 29282 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 01:45:58.181355 29282 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 2 LEADER]: Becoming Leader. State: Replica: 9b0e02ad573e476983ee8c2d623d2501, State: Running, Role: LEADER
I20260809 01:45:58.183060 29282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:58.192243 29283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0e02ad573e476983ee8c2d623d2501. Latest consensus state: current_term: 2 leader_uuid: "9b0e02ad573e476983ee8c2d623d2501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:58.193493 29283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: LEADER
I20260809 01:45:58.195986 29286 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260809 01:45:58.206835 29286 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260809 01:45:58.210656 29286 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:45:58.211087 29286 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260809 01:45:58.219022 29286 catalog_manager.cc:1540] Loading token signing keys...
I20260809 01:45:58.223668 29288 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:45:58.224644 29288 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:58.225407 29286 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Loaded TSK: 0
I20260809 01:45:58.227020 29286 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260809 01:45:58.228641 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:45:58.229077 29288 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:58.229307 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0" is_pre_election: true
I20260809 01:45:58.230031 28647 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9cc5c3257b466c8087bf6fc15b7284 in term 2.
I20260809 01:45:58.232604 29286 catalog_manager.cc:1563] Loading latest processed Hive Metastore notification log event ID...
I20260809 01:45:58.232180 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284; no voters: 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:58.233592 29288 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 01:45:58.234171 29288 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:45:58.234624 29288 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 2 FOLLOWER]: Advancing to term 3
I20260809 01:45:58.239463 29286 catalog_manager.cc:5941] Last processed Hive Metastore notification event ID: -1
I20260809 01:45:58.245687 29288 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:58.248838 29288 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 3 election: Requested vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:45:58.250291 28471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501"
I20260809 01:45:58.250561 28647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "118ef761cca1412eac9223a88e991ff0"
I20260809 01:45:58.251186 28647 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 2 FOLLOWER]: Advancing to term 3
I20260809 01:45:58.259687 28647 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9cc5c3257b466c8087bf6fc15b7284 in term 3.
I20260809 01:45:58.262916 28526 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284; no voters: 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:45:58.264138 29288 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 01:45:58.266034 29288 raft_consensus.cc:697] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 3 LEADER]: Becoming Leader. State: Replica: ba9cc5c3257b466c8087bf6fc15b7284, State: Running, Role: LEADER
I20260809 01:45:58.267102 29288 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:58.274240 29289 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9cc5c3257b466c8087bf6fc15b7284. Latest consensus state: current_term: 3 leader_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:58.275496 29289 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: LEADER
I20260809 01:45:58.280344 29291 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260809 01:45:58.287896 29291 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260809 01:45:58.291720 29291 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:45:58.292160 29291 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260809 01:45:58.299248 29291 catalog_manager.cc:1540] Loading token signing keys...
I20260809 01:45:58.305135 29291 catalog_manager.cc:6103] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Loaded TSK: 0
I20260809 01:45:58.308324 29291 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260809 01:45:58.314041 28461 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56018:
name: "default.Table_ea5a3f9a89e14a24bf4ad0fcb0016253"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:45:58.314395 28460 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56018:
name: "default.Table_6ec1282ccda44a96a1df9fc6f4011570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:45:58.316203 28458 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56018:
name: "default.Table_47e9cb96cf5741709f50572585eaafc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:45:58.316442 28459 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56018:
name: "default.Table_c0467236f2bd4a26863d7f6ec72b5293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:45:58.317284 29291 catalog_manager.cc:1563] Loading latest processed Hive Metastore notification log event ID...
I20260809 01:45:58.322504 29291 catalog_manager.cc:5941] Last processed Hive Metastore notification event ID: -1
W20260809 01:45:58.331418 28461 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ea5a3f9a89e14a24bf4ad0fcb0016253 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260809 01:45:58.345423 28459 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c0467236f2bd4a26863d7f6ec72b5293 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260809 01:45:58.348585 28458 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_47e9cb96cf5741709f50572585eaafc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260809 01:45:58.351197 28460 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6ec1282ccda44a96a1df9fc6f4011570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:45:58.393800 28544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.115.3:39709
I20260809 01:45:58.401396 28457 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.115.3:54097
I20260809 01:45:58.442795 28108 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu with pid 28426
I20260809 01:45:58.521915 28544 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45694:
name: "default.Table_47e9cb96cf5741709f50572585eaafc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:45:58.580122 28544 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_47e9cb96cf5741709f50572585eaafc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260809 01:45:58.588083 28934 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:45:58.659822 28704 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 01:45:58.695782 28529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20260809 01:45:58.711094 28529 consensus_peers.cc:597] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 -> Peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Couldn't send request to peer 9b0e02ad573e476983ee8c2d623d2501. Status: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 01:45:58.850541 29095 heartbeater.cc:499] Master 127.27.115.61:37213 was elected leader, sending a full tablet report...
I20260809 01:45:58.862911 29100 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
W20260809 01:45:58.882262 29100 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:45:58.889241 28647 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Refusing update from remote peer ba9cc5c3257b466c8087bf6fc15b7284: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 01:45:58.893980 29289 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 01:45:58.906075 29265 heartbeater.cc:499] Master 127.27.115.61:37213 was elected leader, sending a full tablet report...
I20260809 01:45:58.908768 29271 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
W20260809 01:45:58.917523 29271 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:45:58.933559 29319 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ba9cc5c3257b466c8087bf6fc15b7284)
I20260809 01:45:58.934480 29319 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:45:58.939121 29317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9cc5c3257b466c8087bf6fc15b7284. Latest consensus state: current_term: 3 leader_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:58.945410 29317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:45:58.945930 28553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:45:58.946462 29319 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
W20260809 01:45:58.971698 28619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260809 01:45:58.989367 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:58.990861 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 01:45:58.991292 28619 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:45:58.991925 28619 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0; no voters: 9b0e02ad573e476983ee8c2d623d2501, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:45:58.993548 29319 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 01:45:58.996871 29289 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:45:58.998191 29289 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: LEADER
I20260809 01:45:59.072278 28930 heartbeater.cc:499] Master 127.27.115.61:37213 was elected leader, sending a full tablet report...
I20260809 01:45:59.327678 28544 catalog_manager.cc:2200] created HMS catalog entry for table [id=a7a29977ecf64312ba9e717a88227c81]
I20260809 01:45:59.376880 28543 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45694:
table { table_name: "default.Table_47e9cb96cf5741709f50572585eaafc1" } new_table_name: "default.Table_749cca3cd8ac4fd3acf868a442b495f4" modify_external_catalogs: true
F20260809 01:45:59.432878 28773 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 2 received events: NotificationEvent(eventId=1, eventTime=1786239959, eventType=CREATE_TABLE, dbName=default, tableName=table_6ec1282ccda44a96a1df9fc6f4011570, message={"server":"","servicePrincipal":"","db":"default","table":"table_6ec1282ccda44a96a1df9fc6f4011570","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6ec1282ccda44a96a1df9fc6f4011570\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1786239958},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/warehouse/table_6ec1282ccda44a96a1df9fc6f4011570\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6ec1282ccda44a96a1df9fc6f4011570\",\"kudu.table_id\":\"773ffabee6f44b739bc87dccbf05b05e\",\"transient_lastDdlTime\":\"1786239958\",\"comment\":\"\",\"kudu.master_addresses\":\"127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e5be15680fc94a8db6a7c4275dbe6560\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1786239959,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1786239959, eventType=CREATE_TABLE, dbName=default, tableName=table_47e9cb96cf5741709f50572585eaafc1, message={"server":"","servicePrincipal":"","db":"default","table":"table_47e9cb96cf5741709f50572585eaafc1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_47e9cb96cf5741709f50572585eaafc1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1786239958},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/warehouse/table_47e9cb96cf5741709f50572585eaafc1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_47e9cb96cf5741709f50572585eaafc1\",\"kudu.table_id\":\"a7a29977ecf64312ba9e717a88227c81\",\"transient_lastDdlTime\":\"1786239958\",\"comment\":\"\",\"kudu.master_addresses\":\"127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e5be15680fc94a8db6a7c4275dbe6560\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1786239959,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260809 01:45:59.463653 29189 tablet_service.cc:1511] Processing CreateTablet for tablet b21f36b7ef324abc94175a43d2664c35 (DEFAULT_TABLE table=default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:45:59.473618 29189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b21f36b7ef324abc94175a43d2664c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:45:59.486342 28853 tablet_service.cc:1511] Processing CreateTablet for tablet b21f36b7ef324abc94175a43d2664c35 (DEFAULT_TABLE table=default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:45:59.489614 28853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b21f36b7ef324abc94175a43d2664c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:45:59.527877 29337 tablet_bootstrap.cc:492] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Bootstrap starting.
I20260809 01:45:59.526881 29019 tablet_service.cc:1511] Processing CreateTablet for tablet b21f36b7ef324abc94175a43d2664c35 (DEFAULT_TABLE table=default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:45:59.531368 29019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b21f36b7ef324abc94175a43d2664c35. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ff9243f9920 google::LogMessage::Flush() at ??:0
I20260809 01:45:59.540727 29337 tablet_bootstrap.cc:654] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Neither blocks nor log segments found. Creating new log.
I20260809 01:45:59.544566 29337 log.cc:826] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Log is configured to *not* fsync() on all Append() calls
I20260809 01:45:59.556623 29337 tablet_bootstrap.cc:492] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: No bootstrap required, opened a new log
I20260809 01:45:59.557776 29337 ts_tablet_manager.cc:1403] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.011s
I20260809 01:45:59.571890 28544 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45694:
name: "default.Table_0bdc257a6d4644109fd42909f87c557d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7ff9243f968f google::LogMessage::~LogMessage() at ??:0
I20260809 01:45:59.588666 29340 tablet_bootstrap.cc:492] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Bootstrap starting.
I20260809 01:45:59.591267 29342 tablet_bootstrap.cc:492] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Bootstrap starting.
I20260809 01:45:59.594156 29337 raft_consensus.cc:359] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:45:59.596540 29337 raft_consensus.cc:385] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:45:59.596939 29337 raft_consensus.cc:740] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b758ff267d534924b3a3204689550643, State: Initialized, Role: FOLLOWER
I20260809 01:45:59.598342 29337 consensus_queue.cc:260] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:45:59.609535 29342 tablet_bootstrap.cc:654] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Neither blocks nor log segments found. Creating new log.
I20260809 01:45:59.614158 29342 log.cc:826] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Log is configured to *not* fsync() on all Append() calls
I20260809 01:45:59.628753 29340 tablet_bootstrap.cc:654] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Neither blocks nor log segments found. Creating new log.
I20260809 01:45:59.630386 29342 tablet_bootstrap.cc:492] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: No bootstrap required, opened a new log
I20260809 01:45:59.631736 29342 ts_tablet_manager.cc:1403] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.020s
I20260809 01:45:59.642284 29337 ts_tablet_manager.cc:1434] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Time spent starting tablet: real 0.084s user 0.040s sys 0.042s
I20260809 01:45:59.680008 29340 log.cc:826] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Log is configured to *not* fsync() on all Append() calls
I20260809 01:45:59.679829 29342 raft_consensus.cc:359] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:45:59.681507 29342 raft_consensus.cc:385] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:45:59.682304 29342 raft_consensus.cc:740] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b148a8b196342d7ade139bdb80e5145, State: Initialized, Role: FOLLOWER
I20260809 01:45:59.684468 29342 consensus_queue.cc:260] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:45:59.696861 29340 tablet_bootstrap.cc:492] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: No bootstrap required, opened a new log
I20260809 01:45:59.697603 29340 ts_tablet_manager.cc:1403] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Time spent bootstrapping tablet: real 0.110s user 0.024s sys 0.000s
I20260809 01:45:59.702045 29342 ts_tablet_manager.cc:1434] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Time spent starting tablet: real 0.069s user 0.056s sys 0.004s
@ 0x7ff92f579fbc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
W20260809 01:45:59.738852 29102 tablet.cc:2406] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:45:59.754570 29340 raft_consensus.cc:359] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
@ 0x7ff92f57977c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260809 01:45:59.755920 29340 raft_consensus.cc:385] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:45:59.756711 29340 raft_consensus.cc:740] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c67f6fb6dd4a8a9c264c5b495c0fc0, State: Initialized, Role: FOLLOWER
I20260809 01:45:59.757838 29340 consensus_queue.cc:260] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
W20260809 01:45:59.860318 29272 tablet.cc:2406] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:45:59.872447 29340 ts_tablet_manager.cc:1434] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Time spent starting tablet: real 0.174s user 0.061s sys 0.046s
@ 0x7ff92f57d7f2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260809 01:45:59.921568 29347 raft_consensus.cc:493] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:45:59.922329 29347 raft_consensus.cc:515] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:45:59.926394 29347 leader_election.cc:290] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405), b758ff267d534924b3a3204689550643 (127.27.115.3:40957)
@ 0x7ff92f57d7aa _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260809 01:45:59.947522 28877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b21f36b7ef324abc94175a43d2664c35" candidate_uuid: "1b148a8b196342d7ade139bdb80e5145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" is_pre_election: true
I20260809 01:45:59.949084 28877 raft_consensus.cc:2468] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b148a8b196342d7ade139bdb80e5145 in term 0.
I20260809 01:45:59.951980 28970 leader_election.cc:304] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b148a8b196342d7ade139bdb80e5145, f4c67f6fb6dd4a8a9c264c5b495c0fc0; no voters:
I20260809 01:45:59.953231 29347 raft_consensus.cc:2804] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:45:59.953673 29347 raft_consensus.cc:493] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:45:59.954030 29347 raft_consensus.cc:3060] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:45:59.956223 29213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b21f36b7ef324abc94175a43d2664c35" candidate_uuid: "1b148a8b196342d7ade139bdb80e5145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b758ff267d534924b3a3204689550643" is_pre_election: true
I20260809 01:45:59.958027 29213 raft_consensus.cc:2468] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b148a8b196342d7ade139bdb80e5145 in term 0.
@ 0x7ff92f57d73a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260809 01:45:59.973629 29347 raft_consensus.cc:515] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:45:59.977160 28877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b21f36b7ef324abc94175a43d2664c35" candidate_uuid: "1b148a8b196342d7ade139bdb80e5145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0"
I20260809 01:45:59.977787 28877 raft_consensus.cc:3060] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:45:59.977916 29347 leader_election.cc:290] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [CANDIDATE]: Term 1 election: Requested vote from peers f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405), b758ff267d534924b3a3204689550643 (127.27.115.3:40957)
I20260809 01:45:59.979400 29213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b21f36b7ef324abc94175a43d2664c35" candidate_uuid: "1b148a8b196342d7ade139bdb80e5145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b758ff267d534924b3a3204689550643"
I20260809 01:45:59.980038 29213 raft_consensus.cc:3060] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:45:59.985639 28877 raft_consensus.cc:2468] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b148a8b196342d7ade139bdb80e5145 in term 1.
I20260809 01:45:59.987205 28970 leader_election.cc:304] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b148a8b196342d7ade139bdb80e5145, f4c67f6fb6dd4a8a9c264c5b495c0fc0; no voters:
I20260809 01:45:59.989066 29213 raft_consensus.cc:2468] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b148a8b196342d7ade139bdb80e5145 in term 1.
I20260809 01:45:59.991506 29347 raft_consensus.cc:2804] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:46:00.008654 29347 raft_consensus.cc:697] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 1 LEADER]: Becoming Leader. State: Replica: 1b148a8b196342d7ade139bdb80e5145, State: Running, Role: LEADER
I20260809 01:46:00.010035 29347 consensus_queue.cc:237] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
@ 0x7ff92f57d702 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260809 01:46:00.037751 28542 catalog_manager.cc:5719] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2). New cstate: current_term: 1 leader_uuid: "1b148a8b196342d7ade139bdb80e5145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } health_report { overall_health: HEALTHY } } }
@ 0x7ff92f57c9fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x5bd8c5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd6fa std::__1::function<>::operator()() at ??:?
@ 0x7ff9265e31eb kudu::Thread::SuperviseThread() at ??:0
@ 0x4568e0 __tsan_thread_start_func at ??:?
@ 0x7ff927d436db start_thread at ??:0
W20260809 01:46:00.104305 28939 tablet.cc:2406] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7ff92309e61f clone at ??:0
W20260809 01:46:00.393257 28618 connection.cc:573] server connection from 127.0.0.1:38078 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:00.393290 29134 connection.cc:573] client connection to 127.27.115.61:37213 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 01:46:00.603557 29379 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ba9cc5c3257b466c8087bf6fc15b7284)
I20260809 01:46:00.604123 29379 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:00.607054 29379 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
W20260809 01:46:00.618800 28619 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:00.619293 28618 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:46:00.619745 28618 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0; no voters: 9b0e02ad573e476983ee8c2d623d2501, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:00.620674 29379 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 01:45:58.509718 29278 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.115.62:33531)
W20260809 01:46:00.394657 29277 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.115.61:37213)
W20260809 01:45:59.803588 28113 debug-util.cc:398] Leaking SignalData structure 0x7b080002d0c0 after lost signal to thread 29278
W20260809 01:46:00.743623 29278 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.27.115.62:33531, user_credentials={real_user=slave}} blocked reactor thread for 2.23585e+06us
W20260809 01:46:00.743883 29277 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.27.115.61:37213, user_credentials={real_user=slave}} blocked reactor thread for 351043us
W20260809 01:46:00.745074 29277 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.27.115.61:37213)
I20260809 01:46:00.745635 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.62:33531
--webserver_interface=127.27.115.62
--webserver_port=35605
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260809 01:46:00.770627 29277 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.27.115.61:37213)
W20260809 01:46:00.865825 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000b44e0 after lost signal to thread 28942
W20260809 01:46:00.866889 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5760 after lost signal to thread 29093
I20260809 01:46:00.908370 29388 consensus_queue.cc:1048] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 01:46:00.951467 29388 consensus_queue.cc:1048] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 01:46:01.264192 29402 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ba9cc5c3257b466c8087bf6fc15b7284)
I20260809 01:46:01.264715 29402 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
W20260809 01:46:01.270906 29385 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:46:01.271996 29385 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:46:01.272893 29385 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260809 01:46:01.273000 29402 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
W20260809 01:46:01.291042 28619 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:01.294421 28618 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:46:01.295460 28618 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0; no voters: 9b0e02ad573e476983ee8c2d623d2501, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:01.296669 29402 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 01:46:01.357295 29385 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:46:01.357820 29385 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:46:01.358228 29385 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:46:01.358659 29385 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:46:01.451182 29385 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.62:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.115.62
--webserver_port=35605
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:46:01.452785 29385 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:46:01.455307 29385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 01:46:01.475595 29408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:46:02.390872 29422 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ba9cc5c3257b466c8087bf6fc15b7284)
I20260809 01:46:02.391964 29422 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:02.408516 29422 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
W20260809 01:46:02.428160 28619 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:02.432601 28618 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:46:02.433817 28618 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0; no voters: 9b0e02ad573e476983ee8c2d623d2501, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:02.435601 29422 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 01:46:01.482545 29410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:46:01.488890 29412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:46:03.243950 29385 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260809 01:46:03.244706 29411 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1758 milliseconds
I20260809 01:46:03.246870 29385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:46:03.251000 29385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:46:03.252781 29385 hybrid_clock.cc:648] HybridClock initialized: now 1786239963252729 us; error 66 us; skew 500 ppm
I20260809 01:46:03.254514 29385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:46:03.265499 29385 webserver.cc:533] Webserver started at http://127.27.115.62:35605/ using document root <none> and password file <none>
I20260809 01:46:03.267621 29385 fs_manager.cc:362] Metadata directory not provided
I20260809 01:46:03.268174 29385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:46:03.281186 29385 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.009s sys 0.000s
I20260809 01:46:03.291345 29434 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:46:03.293887 29385 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.008s sys 0.000s
I20260809 01:46:03.294353 29385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
uuid: "9b0e02ad573e476983ee8c2d623d2501"
format_stamp: "Formatted at 2026-08-09 01:45:40 on dist-test-slave-dhph"
I20260809 01:46:03.297480 29385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:46:03.393625 29385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:46:03.395582 29385 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:46:03.396178 29385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:46:03.524919 29385 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.62:33531
I20260809 01:46:03.524989 29488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.62:33531 every 8 connection(s)
I20260809 01:46:03.528375 29385 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-0/data/info.pb
I20260809 01:46:03.532987 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 29385
I20260809 01:46:03.550613 29489 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:46:03.558764 29489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Bootstrap starting.
I20260809 01:46:03.599437 29489 log.cc:826] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Log is configured to *not* fsync() on all Append() calls
I20260809 01:46:03.601912 28934 heartbeater.cc:344] Connected to a master server at 127.27.115.62:33531
I20260809 01:46:03.628372 29489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 01:46:03.629848 29489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Bootstrap complete.
I20260809 01:46:03.650542 29496 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ba9cc5c3257b466c8087bf6fc15b7284)
I20260809 01:46:03.651188 29496 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:03.655654 29496 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
I20260809 01:46:03.657920 29489 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:03.665390 29489 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0e02ad573e476983ee8c2d623d2501, State: Initialized, Role: FOLLOWER
I20260809 01:46:03.667053 29489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:03.697890 29100 heartbeater.cc:344] Connected to a master server at 127.27.115.62:33531
I20260809 01:46:03.698937 29100 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
I20260809 01:46:03.707365 29499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:03.708488 29499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:03.710443 29489 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 01:46:03.712292 28618 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:46:03.723024 29464 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:46:03.725368 29464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 118ef761cca1412eac9223a88e991ff0 in term 2.
I20260809 01:46:03.727377 28619 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501; no voters: ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:03.728770 29496 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 01:46:03.729137 29496 raft_consensus.cc:493] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ba9cc5c3257b466c8087bf6fc15b7284)
I20260809 01:46:03.729866 29496 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 3 FOLLOWER]: Advancing to term 4
I20260809 01:46:03.740623 29496 raft_consensus.cc:515] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:03.745012 29464 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "118ef761cca1412eac9223a88e991ff0" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501"
I20260809 01:46:03.745929 29464 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 2 FOLLOWER]: Advancing to term 4
I20260809 01:46:03.747138 29496 leader_election.cc:290] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 election: Requested vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213)
W20260809 01:46:03.750944 28618 leader_election.cc:336] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:46:03.757069 29464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 118ef761cca1412eac9223a88e991ff0 in term 4.
I20260809 01:46:03.760650 28619 leader_election.cc:304] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501; no voters: ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:03.761566 29496 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 01:46:03.763275 29496 raft_consensus.cc:697] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [term 4 LEADER]: Becoming Leader. State: Replica: 118ef761cca1412eac9223a88e991ff0, State: Running, Role: LEADER
I20260809 01:46:03.764321 29496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:03.778398 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118ef761cca1412eac9223a88e991ff0. Latest consensus state: current_term: 4 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:03.779882 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:46:03.783757 29519 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260809 01:46:03.832894 29519 catalog_manager.cc:679] Loaded metadata for table default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]
I20260809 01:46:03.840934 29519 tablet_loader.cc:96] loaded metadata for tablet b21f36b7ef324abc94175a43d2664c35 (table default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81])
I20260809 01:46:03.843152 29519 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260809 01:46:03.849562 29519 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:46:03.849987 29519 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260809 01:46:03.856980 29519 catalog_manager.cc:1540] Loading token signing keys...
I20260809 01:46:03.862984 29519 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: Loaded TSK: 0
I20260809 01:46:03.865636 29519 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260809 01:46:03.872036 29519 catalog_manager.cc:1563] Loading latest processed Hive Metastore notification log event ID...
I20260809 01:46:03.876457 29519 catalog_manager.cc:5941] Last processed Hive Metastore notification event ID: -1
I20260809 01:46:03.876667 29523 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:46:03.877223 29523 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: loading cluster ID for follower catalog manager: success
I20260809 01:46:03.900362 29523 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: acquiring CA information for follower catalog manager: success
I20260809 01:46:03.909898 29523 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:46:03.928102 28635 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42360:
name: "default.Table_0bdc257a6d4644109fd42909f87c557d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:46:03.931874 28634 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42360:
table { table_name: "default.Table_47e9cb96cf5741709f50572585eaafc1" } new_table_name: "default.Table_749cca3cd8ac4fd3acf868a442b495f4" modify_external_catalogs: true
I20260809 01:46:03.935839 28633 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42360:
name: "default.Table_c0467236f2bd4a26863d7f6ec72b5293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:46:03.939546 28631 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42360:
name: "default.Table_6ec1282ccda44a96a1df9fc6f4011570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:46:03.943296 28630 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42360:
name: "default.Table_ea5a3f9a89e14a24bf4ad0fcb0016253"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260809 01:46:03.946089 28702 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 2 received events: NotificationEvent(eventId=1, eventTime=1786239959, eventType=CREATE_TABLE, dbName=default, tableName=table_6ec1282ccda44a96a1df9fc6f4011570, message={"server":"","servicePrincipal":"","db":"default","table":"table_6ec1282ccda44a96a1df9fc6f4011570","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6ec1282ccda44a96a1df9fc6f4011570\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1786239958},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/warehouse/table_6ec1282ccda44a96a1df9fc6f4011570\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6ec1282ccda44a96a1df9fc6f4011570\",\"kudu.table_id\":\"773ffabee6f44b739bc87dccbf05b05e\",\"transient_lastDdlTime\":\"1786239958\",\"comment\":\"\",\"kudu.master_addresses\":\"127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e5be15680fc94a8db6a7c4275dbe6560\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1786239959,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1786239959, eventType=CREATE_TABLE, dbName=default, tableName=table_47e9cb96cf5741709f50572585eaafc1, message={"server":"","servicePrincipal":"","db":"default","table":"table_47e9cb96cf5741709f50572585eaafc1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_47e9cb96cf5741709f50572585eaafc1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1786239958},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/warehouse/table_47e9cb96cf5741709f50572585eaafc1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_47e9cb96cf5741709f50572585eaafc1\",\"kudu.table_id\":\"a7a29977ecf64312ba9e717a88227c81\",\"transient_lastDdlTime\":\"1786239958\",\"comment\":\"\",\"kudu.master_addresses\":\"127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e5be15680fc94a8db6a7c4275dbe6560\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1786239959,"files":[]}, messageFormat=json-0.2, catName=hive);
I20260809 01:46:03.946934 28629 master_service.cc:960] ReplaceTablet: received request to replace tablet b21f36b7ef324abc94175a43d2664c35 from {username='slave'} at 127.0.0.1:42360
I20260809 01:46:03.891498 29385 master_runner.cc:187] Error getting master registration for 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111),
I20260809 01:46:03.952790 29385 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:03.948945 28627 master_service.cc:960] ReplaceTablet: received request to replace tablet b21f36b7ef324abc94175a43d2664c35 from {username='slave'} at 127.0.0.1:42360
*** Check failure stack trace: ***
W20260809 01:46:03.975060 28618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20260809 01:46:03.975939 29464 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Refusing update from remote peer 118ef761cca1412eac9223a88e991ff0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20260809 01:46:03.978259 29515 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 01:46:03.979053 29515 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [LEADER]: Peer 9b0e02ad573e476983ee8c2d623d2501 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260809 01:46:03.979497 28618 consensus_peers.cc:597] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 -> Peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Couldn't send request to peer ba9cc5c3257b466c8087bf6fc15b7284. Status: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 01:46:03.986186 29464 pending_rounds.cc:85] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Aborting all ops after (but not including) 4
I20260809 01:46:03.986892 29464 pending_rounds.cc:107] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260809 01:46:03.987668 29464 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7fe75d8d2920 google::LogMessage::Flush() at ??:0
@ 0x7fe75d8d268f google::LogMessage::~LogMessage() at ??:0
I20260809 01:46:04.037704 29271 heartbeater.cc:344] Connected to a master server at 127.27.115.62:33531
I20260809 01:46:04.038317 29271 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
@ 0x7fe768a52fbc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fe768a5277c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260809 01:46:04.119269 29499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118ef761cca1412eac9223a88e991ff0. Latest consensus state: current_term: 4 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:04.120066 29499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:04.111550 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:04.121130 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:04.122720 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:04.123526 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:46:04.124986 29451 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b758ff267d534924b3a3204689550643" instance_seqno: 1786239957726945) as {username='slave'} at 127.27.115.3:56351; Asking this server to re-register.
I20260809 01:46:04.128939 29271 heartbeater.cc:461] Registering TS with master...
I20260809 01:46:04.128693 29452 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" instance_seqno: 1786239951126442) as {username='slave'} at 127.27.115.1:46923; Asking this server to re-register.
I20260809 01:46:04.130748 29499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "118ef761cca1412eac9223a88e991ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:04.131484 29499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:04.131597 28934 heartbeater.cc:461] Registering TS with master...
I20260809 01:46:04.137300 29451 ts_manager.cc:194] Registered new tserver with Master: f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405)
I20260809 01:46:04.135062 29452 ts_manager.cc:194] Registered new tserver with Master: b758ff267d534924b3a3204689550643 (127.27.115.3:40957)
@ 0x7fe768a567f2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260809 01:46:04.143754 28629 catalog_manager.cc:6006] T 00000000000000000000000000000000 P 118ef761cca1412eac9223a88e991ff0: Sending DeleteTablet for 3 replicas of tablet b21f36b7ef324abc94175a43d2664c35
I20260809 01:46:04.150806 29528 mvcc.cc:204] Tried to move back new op lower bound from 7316438892408156160 to 7316438869071413248. Current Snapshot: MvccSnapshot[applied={T|T < 7316438892408156160}]
@ 0x7fe768a567aa _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260809 01:46:04.195578 28629 catalog_manager.cc:6807] ReplaceTablet: tablet b21f36b7ef324abc94175a43d2664c35 deleted and replaced by tablet f29a94779114429c8950a4888ca5d1e9
@ 0x7fe768a5673a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260809 01:46:04.206252 29100 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
I20260809 01:46:04.206881 29189 tablet_service.cc:1558] Processing DeleteTablet for tablet b21f36b7ef324abc94175a43d2664c35 with delete_type TABLET_DATA_DELETED (replaced by tablet f29a94779114429c8950a4888ca5d1e9) from {username='slave'} at 127.0.0.1:56426
I20260809 01:46:04.214696 29452 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" instance_seqno: 1786239954562327) as {username='slave'} at 127.27.115.2:33393; Asking this server to re-register.
I20260809 01:46:04.219914 29100 heartbeater.cc:461] Registering TS with master...
I20260809 01:46:04.225026 29452 ts_manager.cc:194] Registered new tserver with Master: 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2:44481)
@ 0x7fe768a56702 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260809 01:46:04.241016 29547 tablet_replica.cc:333] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: stopping tablet replica
I20260809 01:46:04.262035 29547 raft_consensus.cc:2243] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:46:04.262801 29547 raft_consensus.cc:2272] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fe768a559fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x5bd8c5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd6fa std::__1::function<>::operator()() at ??:?
I20260809 01:46:04.268117 28851 tablet_service.cc:1511] Processing CreateTablet for tablet f29a94779114429c8950a4888ca5d1e9 (DEFAULT_TABLE table=default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:46:04.266196 28853 tablet_service.cc:1558] Processing DeleteTablet for tablet b21f36b7ef324abc94175a43d2664c35 with delete_type TABLET_DATA_DELETED (replaced by tablet f29a94779114429c8950a4888ca5d1e9) from {username='slave'} at 127.0.0.1:39000
I20260809 01:46:04.272082 28851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29a94779114429c8950a4888ca5d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:46:04.276252 29019 tablet_service.cc:1558] Processing DeleteTablet for tablet b21f36b7ef324abc94175a43d2664c35 with delete_type TABLET_DATA_DELETED (replaced by tablet f29a94779114429c8950a4888ca5d1e9) from {username='slave'} at 127.0.0.1:47248
I20260809 01:46:04.277277 29547 ts_tablet_manager.cc:1916] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:46:04.277585 29189 tablet_service.cc:1511] Processing CreateTablet for tablet f29a94779114429c8950a4888ca5d1e9 (DEFAULT_TABLE table=default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:46:04.278371 29017 tablet_service.cc:1511] Processing CreateTablet for tablet f29a94779114429c8950a4888ca5d1e9 (DEFAULT_TABLE table=default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:46:04.279601 29189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29a94779114429c8950a4888ca5d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:46:04.279850 29017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29a94779114429c8950a4888ca5d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:46:04.297840 29552 tablet_replica.cc:333] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: stopping tablet replica
I20260809 01:46:04.298833 29552 raft_consensus.cc:2243] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:46:04.299558 29552 raft_consensus.cc:2272] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:46:04.307617 29554 tablet_bootstrap.cc:492] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643: Bootstrap starting.
I20260809 01:46:04.311168 29553 tablet_replica.cc:333] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: stopping tablet replica
I20260809 01:46:04.314942 29552 ts_tablet_manager.cc:1916] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:46:04.320657 29554 tablet_bootstrap.cc:654] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643: Neither blocks nor log segments found. Creating new log.
@ 0x7fe75fabc1eb kudu::Thread::SuperviseThread() at ??:0
@ 0x4568e0 __tsan_thread_start_func at ??:?
I20260809 01:46:04.325023 29553 raft_consensus.cc:2243] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:46:04.326828 29553 raft_consensus.cc:2272] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:46:04.329473 29557 tablet_bootstrap.cc:492] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Bootstrap starting.
I20260809 01:46:04.337074 29547 ts_tablet_manager.cc:1929] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260809 01:46:04.337649 29547 log.cc:1199] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Deleting WAL directory at /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-2/wal/wals/b21f36b7ef324abc94175a43d2664c35
I20260809 01:46:04.338224 29553 ts_tablet_manager.cc:1916] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:46:04.342444 29559 tablet_bootstrap.cc:492] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145: Bootstrap starting.
I20260809 01:46:04.351143 29547 ts_tablet_manager.cc:1950] T b21f36b7ef324abc94175a43d2664c35 P b758ff267d534924b3a3204689550643: Deleting consensus metadata
I20260809 01:46:04.355377 28618 catalog_manager.cc:5050] TS b758ff267d534924b3a3204689550643 (127.27.115.3:40957): tablet b21f36b7ef324abc94175a43d2664c35 (table default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]) successfully deleted
I20260809 01:46:04.359975 29554 tablet_bootstrap.cc:492] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643: No bootstrap required, opened a new log
I20260809 01:46:04.360450 29554 ts_tablet_manager.cc:1403] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643: Time spent bootstrapping tablet: real 0.054s user 0.020s sys 0.024s
I20260809 01:46:04.370901 28629 master_service.cc:960] ReplaceTablet: received request to replace tablet f29a94779114429c8950a4888ca5d1e9 from {username='slave'} at 127.0.0.1:42360
I20260809 01:46:04.371670 29554 raft_consensus.cc:359] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.372500 29554 raft_consensus.cc:385] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:46:04.372809 29554 raft_consensus.cc:740] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b758ff267d534924b3a3204689550643, State: Initialized, Role: FOLLOWER
I20260809 01:46:04.380853 29557 tablet_bootstrap.cc:654] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Neither blocks nor log segments found. Creating new log.
@ 0x7fe76121c6db start_thread at ??:0
I20260809 01:46:04.391072 29554 consensus_queue.cc:260] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.395543 29559 tablet_bootstrap.cc:654] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145: Neither blocks nor log segments found. Creating new log.
I20260809 01:46:04.400729 29552 ts_tablet_manager.cc:1929] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260809 01:46:04.401463 29552 log.cc:1199] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Deleting WAL directory at /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-0/wal/wals/b21f36b7ef324abc94175a43d2664c35
I20260809 01:46:04.407984 28627 master_service.cc:960] ReplaceTablet: received request to replace tablet f29a94779114429c8950a4888ca5d1e9 from {username='slave'} at 127.0.0.1:42360
@ 0x7fe75c57761f clone at ??:0
I20260809 01:46:04.418756 29559 tablet_bootstrap.cc:492] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145: No bootstrap required, opened a new log
I20260809 01:46:04.419579 29559 ts_tablet_manager.cc:1403] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.011s
I20260809 01:46:04.425495 29553 ts_tablet_manager.cc:1929] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260809 01:46:04.426430 29553 log.cc:1199] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Deleting WAL directory at /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/ts-1/wal/wals/b21f36b7ef324abc94175a43d2664c35
I20260809 01:46:04.426908 29557 tablet_bootstrap.cc:492] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: No bootstrap required, opened a new log
I20260809 01:46:04.427096 29264 heartbeater.cc:499] Master 127.27.115.60:45299 was elected leader, sending a full tablet report...
I20260809 01:46:04.427376 29557 ts_tablet_manager.cc:1403] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Time spent bootstrapping tablet: real 0.098s user 0.013s sys 0.009s
I20260809 01:46:04.427778 29553 ts_tablet_manager.cc:1950] T b21f36b7ef324abc94175a43d2664c35 P 1b148a8b196342d7ade139bdb80e5145: Deleting consensus metadata
I20260809 01:46:04.428254 29563 raft_consensus.cc:493] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:46:04.428723 29563 raft_consensus.cc:515] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.431902 29554 ts_tablet_manager.cc:1434] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643: Time spent starting tablet: real 0.071s user 0.008s sys 0.007s
I20260809 01:46:04.425486 29559 raft_consensus.cc:359] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.442073 29559 raft_consensus.cc:385] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:46:04.442430 29559 raft_consensus.cc:740] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b148a8b196342d7ade139bdb80e5145, State: Initialized, Role: FOLLOWER
I20260809 01:46:04.441169 29563 leader_election.cc:290] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405), 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2:44481)
I20260809 01:46:04.443768 29559 consensus_queue.cc:260] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.445734 29094 heartbeater.cc:499] Master 127.27.115.60:45299 was elected leader, sending a full tablet report...
I20260809 01:46:04.459290 29559 ts_tablet_manager.cc:1434] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145: Time spent starting tablet: real 0.039s user 0.011s sys 0.000s
I20260809 01:46:04.430421 29557 raft_consensus.cc:359] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.470885 29557 raft_consensus.cc:385] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:46:04.471256 29557 raft_consensus.cc:740] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c67f6fb6dd4a8a9c264c5b495c0fc0, State: Initialized, Role: FOLLOWER
I20260809 01:46:04.457468 29552 ts_tablet_manager.cc:1950] T b21f36b7ef324abc94175a43d2664c35 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Deleting consensus metadata
I20260809 01:46:04.481139 29557 consensus_queue.cc:260] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.486500 29045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f29a94779114429c8950a4888ca5d1e9" candidate_uuid: "b758ff267d534924b3a3204689550643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b148a8b196342d7ade139bdb80e5145" is_pre_election: true
I20260809 01:46:04.487434 29045 raft_consensus.cc:2468] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b758ff267d534924b3a3204689550643 in term 0.
I20260809 01:46:04.490094 28877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f29a94779114429c8950a4888ca5d1e9" candidate_uuid: "b758ff267d534924b3a3204689550643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" is_pre_election: true
I20260809 01:46:04.490948 28877 raft_consensus.cc:2468] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b758ff267d534924b3a3204689550643 in term 0.
I20260809 01:46:04.491452 29557 ts_tablet_manager.cc:1434] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0: Time spent starting tablet: real 0.064s user 0.007s sys 0.007s
I20260809 01:46:04.491674 29135 leader_election.cc:304] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b148a8b196342d7ade139bdb80e5145, b758ff267d534924b3a3204689550643; no voters:
I20260809 01:46:04.492226 28929 heartbeater.cc:499] Master 127.27.115.60:45299 was elected leader, sending a full tablet report...
I20260809 01:46:04.493399 29563 raft_consensus.cc:2804] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:46:04.493754 29563 raft_consensus.cc:493] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:46:04.494100 29563 raft_consensus.cc:3060] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:46:04.503728 29563 raft_consensus.cc:515] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
I20260809 01:46:04.508142 29563 leader_election.cc:290] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [CANDIDATE]: Term 1 election: Requested vote from peers f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405), 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2:44481)
I20260809 01:46:04.511924 28877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f29a94779114429c8950a4888ca5d1e9" candidate_uuid: "b758ff267d534924b3a3204689550643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0"
I20260809 01:46:04.513168 28877 raft_consensus.cc:3060] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:46:04.513041 29045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f29a94779114429c8950a4888ca5d1e9" candidate_uuid: "b758ff267d534924b3a3204689550643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b148a8b196342d7ade139bdb80e5145"
I20260809 01:46:04.513866 29045 raft_consensus.cc:3060] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:46:04.522852 28877 raft_consensus.cc:2468] T f29a94779114429c8950a4888ca5d1e9 P f4c67f6fb6dd4a8a9c264c5b495c0fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b758ff267d534924b3a3204689550643 in term 1.
I20260809 01:46:04.523114 29045 raft_consensus.cc:2468] T f29a94779114429c8950a4888ca5d1e9 P 1b148a8b196342d7ade139bdb80e5145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b758ff267d534924b3a3204689550643 in term 1.
I20260809 01:46:04.524807 29135 leader_election.cc:304] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b758ff267d534924b3a3204689550643, f4c67f6fb6dd4a8a9c264c5b495c0fc0; no voters:
I20260809 01:46:04.526396 29563 raft_consensus.cc:2804] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:46:04.530793 29563 raft_consensus.cc:697] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [term 1 LEADER]: Becoming Leader. State: Replica: b758ff267d534924b3a3204689550643, State: Running, Role: LEADER
I20260809 01:46:04.533283 29563 consensus_queue.cc:237] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 } } peers { permanent_uuid: "b758ff267d534924b3a3204689550643" member_type: VOTER last_known_addr { host: "127.27.115.3" port: 40957 } } peers { permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 } }
W20260809 01:46:04.795389 28966 connection.cc:573] server connection from 127.0.0.1:47248 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:04.796103 28966 connection.cc:573] client connection to 127.27.115.60:45299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:04.797298 29131 connection.cc:573] client connection to 127.27.115.60:45299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:04.797595 29275 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.27.115.60:45299)
W20260809 01:46:04.798851 29275 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.115.60:45299)
W20260809 01:46:04.800441 29275 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.27.115.60:45299)
W20260809 01:46:04.800355 28797 connection.cc:573] server connection from 127.0.0.1:39000 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:04.800933 28797 connection.cc:573] client connection to 127.27.115.60:45299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:04.805020 29438 connection.cc:573] server connection from 127.0.0.1:56740 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 01:46:04.915729 29576 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:46:04.916262 29576 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:04.919319 29576 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:04.922607 29437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
W20260809 01:46:04.930994 29437 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
W20260809 01:46:04.931519 29435 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:04.931974 29435 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:04.932972 29576 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 01:46:05.445197 29582 consensus_queue.cc:1048] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" member_type: VOTER last_known_addr { host: "127.27.115.2" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 01:46:05.490476 29582 consensus_queue.cc:1048] T f29a94779114429c8950a4888ca5d1e9 P b758ff267d534924b3a3204689550643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" member_type: VOTER last_known_addr { host: "127.27.115.1" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 01:46:05.578789 29595 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:46:05.579309 29595 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:05.581204 29595 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:05.587360 29437 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
W20260809 01:46:05.587882 29435 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:05.588310 29435 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:05.589493 29595 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 01:46:06.557762 29606 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:46:06.558240 29606 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:06.560599 29606 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:06.568917 29437 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
W20260809 01:46:06.569818 29435 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:06.570711 29435 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:06.571619 29606 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 01:46:07.209259 28108 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
/tmp/dist-test-taskFVJal6/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.61:37213
--webserver_interface=127.27.115.61
--webserver_port=42297
--builtin_ntp_servers=127.27.115.20:39971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260809 01:46:07.614372 29611 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:46:07.615149 29611 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:46:07.615669 29611 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 01:46:07.653508 29611 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:46:07.654047 29611 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:46:07.654511 29611 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:46:07.654920 29611 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:46:07.695904 29611 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.115.20:39971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45531
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.115.61:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.115.61
--webserver_port=42297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type FASTDEBUG
built by None at 09 Aug 2026 01:43:22 UTC on e7f111948823
build id 13251
TSAN enabled
I20260809 01:46:07.697443 29611 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:46:07.699421 29611 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 01:46:07.714664 29620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:46:08.211757 29630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:46:08.214185 29630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:08.249320 29630 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:08.312546 29435 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:46:08.319474 29437 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213): Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111)
I20260809 01:46:08.320959 29437 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501; no voters: 118ef761cca1412eac9223a88e991ff0, ba9cc5c3257b466c8087bf6fc15b7284
I20260809 01:46:08.322450 29630 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 01:46:09.030462 29385 master_runner.cc:187] Error getting master registration for 127.27.115.61:37213: Network error: Client connection negotiation failed: client connection to 127.27.115.61:37213: connect: Connection refused (error 111),
I20260809 01:46:09.031489 29385 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:46:07.715691 29621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:46:09.119057 29619 debug-util.cc:398] Leaking SignalData structure 0x7b08000374a0 after lost signal to thread 29611
W20260809 01:46:09.619130 29622 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1901 milliseconds
W20260809 01:46:09.617429 29611 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.903s user 0.656s sys 1.192s
W20260809 01:46:09.620438 29611 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.906s user 0.657s sys 1.192s
I20260809 01:46:09.620790 29611 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260809 01:46:09.619272 29624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:46:09.632388 29611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:46:09.639251 29611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:46:09.640908 29611 hybrid_clock.cc:648] HybridClock initialized: now 1786239969640841 us; error 76 us; skew 500 ppm
I20260809 01:46:09.642443 29611 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 01:46:09.652822 29611 webserver.cc:533] Webserver started at http://127.27.115.61:42297/ using document root <none> and password file <none>
I20260809 01:46:09.655208 29611 fs_manager.cc:362] Metadata directory not provided
I20260809 01:46:09.655759 29611 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:46:09.674414 29611 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.005s sys 0.005s
I20260809 01:46:09.681917 29647 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:46:09.683564 29611 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.000s sys 0.006s
I20260809 01:46:09.684113 29611 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data,/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
format_stamp: "Formatted at 2026-08-09 01:45:43 on dist-test-slave-dhph"
I20260809 01:46:09.687371 29611 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:46:09.753696 29611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 01:46:09.756099 29611 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:46:09.756822 29611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:46:09.853238 29611 rpc_server.cc:307] RPC server started. Bound to: 127.27.115.61:37213
I20260809 01:46:09.853534 29704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.115.61:37213 every 8 connection(s)
I20260809 01:46:09.856765 29611 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/master-1/data/info.pb
I20260809 01:46:09.861296 28108 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskFVJal6/build/tsan/bin/kudu as pid 29611
I20260809 01:46:09.875469 29705 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:46:09.885911 29705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Bootstrap starting.
I20260809 01:46:09.903616 29705 log.cc:826] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Log is configured to *not* fsync() on all Append() calls
I20260809 01:46:09.936641 29705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=1 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 01:46:09.937956 29705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: Bootstrap complete.
I20260809 01:46:09.965345 29705 raft_consensus.cc:359] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:09.971273 29265 heartbeater.cc:344] Connected to a master server at 127.27.115.61:37213
I20260809 01:46:09.978668 29705 raft_consensus.cc:740] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9cc5c3257b466c8087bf6fc15b7284, State: Initialized, Role: FOLLOWER
I20260809 01:46:09.979713 29705 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:09.982517 29711 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:09.983495 29711 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:09.984759 29705 sys_catalog.cc:565] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:46:10.033457 29728 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:46:10.033910 29728 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: loading cluster ID for follower catalog manager: success
I20260809 01:46:10.054504 29728 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: acquiring CA information for follower catalog manager: success
I20260809 01:46:10.061259 29728 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:46:10.079843 29095 heartbeater.cc:344] Connected to a master server at 127.27.115.61:37213
I20260809 01:46:10.092113 29665 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b148a8b196342d7ade139bdb80e5145" instance_seqno: 1786239954562327) as {username='slave'} at 127.27.115.2:47281; Asking this server to re-register.
I20260809 01:46:10.094425 29095 heartbeater.cc:461] Registering TS with master...
I20260809 01:46:10.100148 29664 ts_manager.cc:194] Registered new tserver with Master: 1b148a8b196342d7ade139bdb80e5145 (127.27.115.2:44481)
I20260809 01:46:10.144718 28930 heartbeater.cc:344] Connected to a master server at 127.27.115.61:37213
I20260809 01:46:10.150553 29665 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f4c67f6fb6dd4a8a9c264c5b495c0fc0" instance_seqno: 1786239951126442) as {username='slave'} at 127.27.115.1:41507; Asking this server to re-register.
I20260809 01:46:10.152508 28930 heartbeater.cc:461] Registering TS with master...
I20260809 01:46:10.156800 29665 ts_manager.cc:194] Registered new tserver with Master: f4c67f6fb6dd4a8a9c264c5b495c0fc0 (127.27.115.1:40405)
I20260809 01:46:10.114705 29611 master_runner.cc:187] Error getting master registration for 127.27.115.60:45299: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111),
I20260809 01:46:10.171986 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:10.384377 29736 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:46:10.384856 29736 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:10.387180 29736 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:10.392567 29435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20260809 01:46:10.397030 29435 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:10.400962 29678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" is_pre_election: true
I20260809 01:46:10.401924 29678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0e02ad573e476983ee8c2d623d2501 in term 3.
I20260809 01:46:10.403393 29437 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501, ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0
I20260809 01:46:10.404150 29736 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 01:46:10.404501 29736 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 118ef761cca1412eac9223a88e991ff0)
I20260809 01:46:10.404846 29736 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 4 FOLLOWER]: Advancing to term 5
I20260809 01:46:10.410360 29736 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:10.412314 29736 leader_election.cc:290] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 election: Requested vote from peers ba9cc5c3257b466c8087bf6fc15b7284 (127.27.115.61:37213), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
I20260809 01:46:10.413288 29678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b0e02ad573e476983ee8c2d623d2501" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "ba9cc5c3257b466c8087bf6fc15b7284"
I20260809 01:46:10.414175 29678 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 3 FOLLOWER]: Advancing to term 5
W20260809 01:46:10.418396 29435 leader_election.cc:336] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:10.422537 29678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0e02ad573e476983ee8c2d623d2501 in term 5.
I20260809 01:46:10.424014 29437 leader_election.cc:304] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b0e02ad573e476983ee8c2d623d2501, ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0
I20260809 01:46:10.424770 29736 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 01:46:10.426421 29736 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [term 5 LEADER]: Becoming Leader. State: Replica: 9b0e02ad573e476983ee8c2d623d2501, State: Running, Role: LEADER
I20260809 01:46:10.427613 29736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:10.435187 29741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0e02ad573e476983ee8c2d623d2501. Latest consensus state: current_term: 5 leader_uuid: "9b0e02ad573e476983ee8c2d623d2501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:10.439509 29741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: LEADER
I20260809 01:46:10.482961 29665 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b758ff267d534924b3a3204689550643" instance_seqno: 1786239957726945) as {username='slave'} at 127.27.115.3:60947; Asking this server to re-register.
I20260809 01:46:10.484843 29265 heartbeater.cc:461] Registering TS with master...
I20260809 01:46:10.490101 29665 ts_manager.cc:194] Registered new tserver with Master: b758ff267d534924b3a3204689550643 (127.27.115.3:40957)
I20260809 01:46:10.857515 29678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Refusing update from remote peer 9b0e02ad573e476983ee8c2d623d2501: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
I20260809 01:46:10.860117 29741 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 01:46:10.903053 29748 mvcc.cc:204] Tried to move back new op lower bound from 7316438893328441344 to 7316438891590995968. Current Snapshot: MvccSnapshot[applied={T|T < 7316438893328441344 or (T in {7316438893328441344})}]
I20260809 01:46:10.913173 29746 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0e02ad573e476983ee8c2d623d2501. Latest consensus state: current_term: 5 leader_uuid: "9b0e02ad573e476983ee8c2d623d2501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:10.914698 29746 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:10.916121 29751 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "9b0e02ad573e476983ee8c2d623d2501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:10.917183 29751 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:10.922781 29751 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "9b0e02ad573e476983ee8c2d623d2501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:10.923703 29751 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:46:10.930899 29736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "9b0e02ad573e476983ee8c2d623d2501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } } }
I20260809 01:46:10.931787 29736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 [sys.catalog]: This master's current role is: LEADER
W20260809 01:46:10.942719 29435 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501 -> Peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Couldn't send request to peer 118ef761cca1412eac9223a88e991ff0. Status: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 01:46:10.962312 29743 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260809 01:46:10.971745 29743 catalog_manager.cc:679] Loaded metadata for table default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81]
I20260809 01:46:10.982465 29743 tablet_loader.cc:96] loaded metadata for tablet e87354d6d6864322908168df02808bdc (table default.table_47e9cb96cf5741709f50572585eaafc1 [id=a7a29977ecf64312ba9e717a88227c81])
I20260809 01:46:10.985183 29743 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260809 01:46:10.991097 29743 catalog_manager.cc:1295] Loaded cluster ID: e5be15680fc94a8db6a7c4275dbe6560
I20260809 01:46:10.991451 29743 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260809 01:46:10.999233 29743 catalog_manager.cc:1540] Loading token signing keys...
I20260809 01:46:11.007653 29743 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 9b0e02ad573e476983ee8c2d623d2501: Loaded TSK: 0
I20260809 01:46:11.009894 29743 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260809 01:46:11.018909 29743 catalog_manager.cc:1563] Loading latest processed Hive Metastore notification log event ID...
I20260809 01:46:11.025890 29743 catalog_manager.cc:5941] Last processed Hive Metastore notification event ID: -1
I20260809 01:46:11.101161 29451 master_service.cc:960] ReplaceTablet: received request to replace tablet f29a94779114429c8950a4888ca5d1e9 from {username='slave'} at 127.0.0.1:56732
I20260809 01:46:11.101724 29452 master_service.cc:960] ReplaceTablet: received request to replace tablet f29a94779114429c8950a4888ca5d1e9 from {username='slave'} at 127.0.0.1:56732
I20260809 01:46:11.102267 29450 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "default.Table_ea5a3f9a89e14a24bf4ad0fcb0016253"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:46:11.105381 29449 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "default.Table_6ec1282ccda44a96a1df9fc6f4011570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:46:11.108925 29448 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "default.Table_c0467236f2bd4a26863d7f6ec72b5293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:46:11.110796 29447 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56732:
table { table_name: "default.Table_47e9cb96cf5741709f50572585eaafc1" } new_table_name: "default.Table_749cca3cd8ac4fd3acf868a442b495f4" modify_external_catalogs: true
I20260809 01:46:11.112643 29446 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "default.Table_0bdc257a6d4644109fd42909f87c557d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260809 01:46:11.125216 29520 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 2 received events: NotificationEvent(eventId=1, eventTime=1786239959, eventType=CREATE_TABLE, dbName=default, tableName=table_6ec1282ccda44a96a1df9fc6f4011570, message={"server":"","servicePrincipal":"","db":"default","table":"table_6ec1282ccda44a96a1df9fc6f4011570","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6ec1282ccda44a96a1df9fc6f4011570\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1786239958},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/warehouse/table_6ec1282ccda44a96a1df9fc6f4011570\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6ec1282ccda44a96a1df9fc6f4011570\",\"kudu.table_id\":\"773ffabee6f44b739bc87dccbf05b05e\",\"transient_lastDdlTime\":\"1786239958\",\"comment\":\"\",\"kudu.master_addresses\":\"127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e5be15680fc94a8db6a7c4275dbe6560\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1786239959,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1786239959, eventType=CREATE_TABLE, dbName=default, tableName=table_47e9cb96cf5741709f50572585eaafc1, message={"server":"","servicePrincipal":"","db":"default","table":"table_47e9cb96cf5741709f50572585eaafc1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_47e9cb96cf5741709f50572585eaafc1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1786239958},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskFVJal6/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1786239916107437-28108-0/minicluster-data/warehouse/table_47e9cb96cf5741709f50572585eaafc1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_47e9cb96cf5741709f50572585eaafc1\",\"kudu.table_id\":\"a7a29977ecf64312ba9e717a88227c81\",\"transient_lastDdlTime\":\"1786239958\",\"comment\":\"\",\"kudu.master_addresses\":\"127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e5be15680fc94a8db6a7c4275dbe6560\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1786239959,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f2a43007920 google::LogMessage::Flush() at ??:0
@ 0x7f2a4300768f google::LogMessage::~LogMessage() at ??:0
W20260809 01:46:11.196959 29485 debug-util.cc:398] Leaking SignalData structure 0x7b0800070060 after lost signal to thread 29398
W20260809 01:46:11.197799 29485 debug-util.cc:398] Leaking SignalData structure 0x7b08000700a0 after lost signal to thread 29488
@ 0x7f2a4e187fbc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f2a4e18777c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260809 01:46:11.318476 29452 master_service.cc:960] ReplaceTablet: received request to replace tablet f29a94779114429c8950a4888ca5d1e9 from {username='slave'} at 127.0.0.1:56732
I20260809 01:46:11.321401 29451 master_service.cc:960] ReplaceTablet: received request to replace tablet f29a94779114429c8950a4888ca5d1e9 from {username='slave'} at 127.0.0.1:56732
@ 0x7f2a4e18b7f2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f2a4e18b7aa _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f2a4e18b73a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f2a4e18b702 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260809 01:46:11.417131 29751 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:11.417712 29751 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:11.425683 29464 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" candidate_term: 6 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: false dest_uuid: "9b0e02ad573e476983ee8c2d623d2501" is_pre_election: true
I20260809 01:46:11.428344 29751 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
@ 0x7f2a4e18a9fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x5bd8c5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd6fa std::__1::function<>::operator()() at ??:?
W20260809 01:46:11.450537 29648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:46:11.464901 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:11.465677 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:11.466751 29751 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
@ 0x7f2a451f11eb kudu::Thread::SuperviseThread() at ??:0
@ 0x4568e0 __tsan_thread_start_func at ??:?
@ 0x7f2a469516db start_thread at ??:0
@ 0x7f2a41cac61f clone at ??:0
I20260809 01:46:11.557432 29271 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
I20260809 01:46:11.566347 29100 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
I20260809 01:46:11.635422 28934 heartbeater.cc:499] Master 127.27.115.62:33531 was elected leader, sending a full tablet report...
W20260809 01:46:11.800325 29135 connection.cc:573] client connection to 127.27.115.62:33531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:11.802531 28970 connection.cc:573] client connection to 127.27.115.62:33531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:11.802611 28801 connection.cc:573] client connection to 127.27.115.62:33531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:11.803936 29278 connection.cc:573] client connection to 127.27.115.62:33531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:46:11.805922 29278 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.27.115.62:33531)
W20260809 01:46:11.815249 29278 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.115.62:33531)
W20260809 01:46:11.816422 29278 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.27.115.62:33531)
I20260809 01:46:12.028445 29783 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:12.029135 29783 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:12.042518 29783 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:12.064532 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:12.065084 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:12.065476 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:12.066238 29783 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:12.456799 28108 master-stress-test.cc:579] Killing test thread 1/10
I20260809 01:46:13.188774 29790 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:13.189260 29790 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:13.192885 29790 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:13.199117 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:13.200711 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:13.201184 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:13.202062 29790 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:14.873224 29795 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:14.873733 29795 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:14.875780 29795 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:14.882901 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:14.883677 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:14.884091 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:14.884835 29795 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:15.175999 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:15.176417 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:16.303112 29802 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:16.303625 29802 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:16.305374 29802 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:16.313604 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:16.314229 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:16.314709 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:16.315392 29802 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:19.304530 29814 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:19.305296 29814 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:19.308195 29814 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:19.312369 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260809 01:46:19.316378 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:19.317008 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:19.317366 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:19.318074 29814 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:20.180195 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:20.180786 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:46:20.554697 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000421c0 after lost signal to thread 29108
W20260809 01:46:20.556169 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca300 after lost signal to thread 29263
I20260809 01:46:24.365254 29826 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:24.365705 29826 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:24.367933 29826 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:24.370426 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:46:24.373605 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:46:24.374213 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:46:24.374894 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:24.375694 29826 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:25.184633 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:25.185478 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:30.192302 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:30.193390 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:32.256897 29850 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:32.257562 29850 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:32.259817 29850 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:32.262668 29648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:46:32.266510 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:32.267390 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:46:32.267797 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:32.269282 29850 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:35.197949 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:35.198521 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:38.999681 29867 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:39.000545 29867 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:39.002915 29867 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:39.010814 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:46:39.016136 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:46:39.016922 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:46:39.017743 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:39.018656 29867 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:46:40.206008 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:40.206614 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:44.095635 29881 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:46:44.096197 29881 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:46:44.098554 29881 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:46:44.101598 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:46:44.105547 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:46:44.106088 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:46:44.106489 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:46:44.107599 29881 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 01:46:44.165023 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d56a0 after lost signal to thread 29108
W20260809 01:46:44.166561 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d17c0 after lost signal to thread 29263
I20260809 01:46:45.211787 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:45.212376 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:50.217996 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:50.218811 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:46:55.223040 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:46:55.223466 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:46:58.648311 28934 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (93 consecutive failures): Network error: Failed to ping master at 127.27.115.62:33531: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:46:58.979135 29094 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (110 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:46:59.146015 29271 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (94 consecutive failures): Network error: Failed to ping master at 127.27.115.62:33531: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:47:00.228169 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:00.228991 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:00.388494 29914 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:47:00.389209 29914 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:47:00.392315 29914 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:47:00.393966 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:47:00.399437 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:47:00.400069 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:47:00.400416 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:47:00.401114 29914 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:47:05.233989 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:05.234553 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:47:05.701936 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2f60 after lost signal to thread 28942
W20260809 01:47:05.703056 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2080 after lost signal to thread 29093
I20260809 01:47:10.239724 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:10.240219 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:15.243991 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:15.244429 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:17.049664 29947 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:47:17.050410 29947 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:47:17.053963 29947 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:47:17.057813 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:47:17.063714 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:47:17.064500 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:47:17.064874 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:47:17.065881 29947 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 01:47:17.551800 28925 debug-util.cc:398] Leaking SignalData structure 0x7b0800041780 after lost signal to thread 28740
W20260809 01:47:17.552909 28925 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb820 after lost signal to thread 28928
I20260809 01:47:20.249907 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:20.250986 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:20.331269 29959 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:47:20.331918 29959 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:47:20.334077 29959 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:47:20.340374 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:47:20.341111 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:47:20.341485 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:47:20.342231 29959 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:47:25.255694 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:25.256268 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:27.888293 29979 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:47:27.889312 29979 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:47:27.892475 29979 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:47:27.907672 29648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260809 01:47:27.914165 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:47:27.915124 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:47:27.915778 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:47:27.918735 29979 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:47:30.260181 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:30.260731 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:35.264953 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:35.265393 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:47:36.935562 29701 debug-util.cc:398] Leaking SignalData structure 0x7b0800099740 after lost signal to thread 29615
W20260809 01:47:36.936185 29701 debug-util.cc:398] Leaking SignalData structure 0x7b0800070f40 after lost signal to thread 29704
I20260809 01:47:40.319700 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:40.320516 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:42.252153 30011 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:47:42.253002 30011 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:47:42.256119 30011 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:47:42.262524 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:47:42.269263 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:47:42.270442 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:47:42.271198 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:47:42.272778 30011 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:47:45.324807 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:45.325212 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:50.329990 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:50.330667 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:47:51.324903 30036 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:47:51.325790 30036 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:47:51.328507 30036 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:47:51.331760 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:47:51.337525 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:47:51.338472 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:47:51.341123 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:47:51.342509 30036 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 01:47:51.522343 29701 debug-util.cc:398] Leaking SignalData structure 0x7b08000704c0 after lost signal to thread 29615
W20260809 01:47:51.523983 29701 debug-util.cc:398] Leaking SignalData structure 0x7b0800042540 after lost signal to thread 29704
I20260809 01:47:55.386838 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:47:55.387351 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:47:58.867364 28929 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (229 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:47:59.180346 29100 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (213 consecutive failures): Network error: Failed to ping master at 127.27.115.62:33531: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:47:59.232815 29264 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (229 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:48:00.392695 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:00.393507 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:48:03.802770 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdb60 after lost signal to thread 29108
W20260809 01:48:03.804186 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdba0 after lost signal to thread 29263
I20260809 01:48:05.398093 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:05.398860 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:48:09.991230 29653 maintenance_manager.cc:419] P ba9cc5c3257b466c8087bf6fc15b7284: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033336
I20260809 01:48:10.094973 29652 maintenance_manager.cc:643] P ba9cc5c3257b466c8087bf6fc15b7284: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.102s user 0.058s sys 0.020s Metrics: {"bytes_written":7101,"cfile_init":1,"dirs.queue_time_us":913,"dirs.run_cpu_time_us":472,"dirs.run_wall_time_us":1189,"drs_written":1,"lbm_read_time_us":312,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2870,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":7,"thread_start_us":996,"threads_started":2,"wal-append.queue_time_us":802}
I20260809 01:48:10.098481 29653 maintenance_manager.cc:419] P ba9cc5c3257b466c8087bf6fc15b7284: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 597 bytes on disk
I20260809 01:48:10.100977 29652 maintenance_manager.cc:643] P ba9cc5c3257b466c8087bf6fc15b7284: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":397,"lbm_reads_lt_1ms":4}
I20260809 01:48:10.405169 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:10.405799 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:48:10.757341 30083 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:48:10.758214 30083 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:48:10.762216 30083 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:48:10.764760 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:48:10.771320 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:48:10.772331 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:48:10.772768 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:48:10.773629 30083 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:48:13.463369 30091 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:48:13.463981 30091 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:48:13.466271 30091 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:48:13.474267 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:48:13.476156 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:48:13.476608 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:48:13.477419 30091 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:48:15.409539 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:15.410146 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:48:20.415025 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:20.415702 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:48:22.258314 28925 debug-util.cc:398] Leaking SignalData structure 0x7b0800042860 after lost signal to thread 28740
W20260809 01:48:22.259696 28925 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcdc0 after lost signal to thread 28928
W20260809 01:48:25.198716 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000b67c0 after lost signal to thread 29108
W20260809 01:48:25.200093 29259 debug-util.cc:398] Leaking SignalData structure 0x7b0800041de0 after lost signal to thread 29263
I20260809 01:48:25.419639 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:25.420186 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:48:27.904410 30162 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:48:27.905041 30162 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:48:27.907775 30162 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:48:27.911402 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260809 01:48:27.915082 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:48:27.915694 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:48:27.916234 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:48:27.917095 30162 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:48:30.429430 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:30.430037 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:48:35.434250 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:35.434793 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:48:35.461798 29259 debug-util.cc:398] Leaking SignalData structure 0x7b080003ed20 after lost signal to thread 29108
W20260809 01:48:35.462935 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d12a0 after lost signal to thread 29263
I20260809 01:48:38.826448 30232 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:48:38.827215 30232 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:48:38.829347 30232 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:48:38.833235 29648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:48:38.837579 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:48:38.838214 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:48:38.838518 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:48:38.839236 30232 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:48:40.439306 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:40.439864 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:48:45.444341 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:45.444917 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:48:49.011471 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0060 after lost signal to thread 29108
W20260809 01:48:49.012723 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5fa0 after lost signal to thread 29263
I20260809 01:48:50.449250 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:50.449779 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:48:53.060355 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000bca20 after lost signal to thread 28942
W20260809 01:48:53.061681 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000b96a0 after lost signal to thread 29093
I20260809 01:48:53.750670 30320 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:48:53.751214 30320 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:48:53.753403 30320 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:48:53.755787 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:48:53.759819 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:48:53.760488 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:48:53.760776 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:48:53.761353 30320 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 01:48:54.353899 28925 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8800 after lost signal to thread 28740
W20260809 01:48:54.354969 28925 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdd60 after lost signal to thread 28928
I20260809 01:48:55.453230 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:48:55.453608 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:48:56.825004 30339 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:48:56.825623 30339 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:48:56.828270 30339 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:48:56.834291 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:48:56.835182 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:48:56.835554 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:48:56.836468 30339 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 01:48:57.037889 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000d29a0 after lost signal to thread 28942
W20260809 01:48:57.039347 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5f00 after lost signal to thread 29093
W20260809 01:48:58.912434 28934 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (332 consecutive failures): Network error: Failed to ping master at 127.27.115.62:33531: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:48:59.256363 29094 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (349 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:48:59.306095 29264 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (348 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:49:00.457823 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:00.458463 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:05.462275 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:05.463117 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:10.468269 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:10.468804 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:49:11.076678 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d11e0 after lost signal to thread 29108
W20260809 01:49:11.077652 29259 debug-util.cc:398] Leaking SignalData structure 0x7b0800041360 after lost signal to thread 29263
W20260809 01:49:12.147646 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc6c0 after lost signal to thread 29108
W20260809 01:49:12.149037 29259 debug-util.cc:398] Leaking SignalData structure 0x7b080003ee80 after lost signal to thread 29263
I20260809 01:49:15.473021 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:15.473429 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:16.214742 30449 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:49:16.215307 30449 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:49:16.217645 30449 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:49:16.220268 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260809 01:49:16.224068 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:49:16.224594 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:49:16.225342 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:49:16.226225 30449 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 01:49:20.372385 29701 debug-util.cc:398] Leaking SignalData structure 0x7b0800070a40 after lost signal to thread 29615
W20260809 01:49:20.373128 29701 debug-util.cc:398] Leaking SignalData structure 0x7b0800070260 after lost signal to thread 29704
I20260809 01:49:20.515498 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:20.516019 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:25.519863 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:25.520422 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:26.943966 30510 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:49:26.944962 30510 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:49:26.947901 30510 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:49:26.950356 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:49:26.954833 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:49:26.955387 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:49:26.955755 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:49:26.956526 30510 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:49:30.525802 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:30.526814 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:35.532630 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:35.533051 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:40.537993 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:40.538712 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:43.096827 30598 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:49:43.097522 30598 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:49:43.099967 30598 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:49:43.102735 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:49:43.106529 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:49:43.107542 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:49:43.107975 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:49:43.109179 30598 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:49:45.543439 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:45.543994 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:50.548888 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:50.549708 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:49:51.951511 30647 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:49:51.952113 30647 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:49:51.955358 30647 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:49:51.958393 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:49:51.961679 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:49:51.962335 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:49:51.963048 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:49:51.963871 30647 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 01:49:54.808779 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca2e0 after lost signal to thread 28942
W20260809 01:49:54.810247 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7e20 after lost signal to thread 29093
I20260809 01:49:55.553320 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:49:55.553938 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:49:59.138021 28929 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (468 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:49:59.462989 29100 heartbeater.cc:646] Failed to heartbeat to 127.27.115.62:33531 (452 consecutive failures): Network error: Failed to ping master at 127.27.115.62:33531: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:49:59.663470 29264 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (468 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:50:00.559090 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:00.559938 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:04.217644 30713 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:50:04.218446 30713 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:50:04.220415 30713 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:50:04.223469 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:50:04.227826 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:50:04.228717 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:50:04.229600 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:50:04.230472 30713 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:50:05.563755 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:05.564263 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:50:08.267468 29701 debug-util.cc:398] Leaking SignalData structure 0x7b0800099100 after lost signal to thread 29615
W20260809 01:50:08.268587 29701 debug-util.cc:398] Leaking SignalData structure 0x7b0800070060 after lost signal to thread 29704
I20260809 01:50:10.598002 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:10.598948 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:11.008821 30750 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:50:11.009814 30750 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:50:11.013284 30750 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:50:11.014912 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:50:11.019682 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:50:11.020565 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:50:11.020956 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:50:11.022195 30750 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:50:11.927435 30759 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:50:11.928002 30759 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:50:11.930373 30759 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:50:11.935847 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:50:11.936389 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:50:11.936730 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:50:11.937398 30759 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:50:15.602806 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:15.603211 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:19.683750 30801 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:50:19.684403 30801 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:50:19.686995 30801 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:50:19.691005 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:50:19.690999 29648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260809 01:50:19.694667 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:50:19.695221 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:50:19.695519 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:50:19.696533 30801 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:50:20.608661 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:20.609514 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:25.612921 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:25.613348 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:30.618523 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:30.619200 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:33.785665 30874 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:50:33.786237 30874 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:50:33.788555 30874 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:50:33.790905 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:50:33.795200 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:50:33.795943 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
I20260809 01:50:33.796360 29651 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:50:33.797166 30874 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:50:35.624346 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:35.624787 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:40.629983 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:40.630877 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:50:44.247550 28925 debug-util.cc:398] Leaking SignalData structure 0x7b08000ced20 after lost signal to thread 28740
W20260809 01:50:44.248834 28925 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac560 after lost signal to thread 28928
I20260809 01:50:45.635043 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:45.635468 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:48.259106 30948 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:50:48.259969 30948 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:50:48.262768 30948 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:50:48.265805 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:50:48.271216 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:50:48.272163 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:50:48.272575 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:50:48.273443 30948 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:50:50.640328 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:50.641008 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:50:55.644627 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:50:55.645316 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:50:55.750875 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5a40 after lost signal to thread 29108
W20260809 01:50:55.751850 29259 debug-util.cc:398] Leaking SignalData structure 0x7b08000d00c0 after lost signal to thread 29263
W20260809 01:50:55.907243 29090 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca1e0 after lost signal to thread 28942
W20260809 01:50:55.908530 29090 debug-util.cc:398] Leaking SignalData structure 0x7b0800041b20 after lost signal to thread 29093
W20260809 01:50:59.484661 29094 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (588 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:50:59.493734 28929 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (588 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
W20260809 01:51:00.013386 29264 heartbeater.cc:646] Failed to heartbeat to 127.27.115.60:45299 (588 consecutive failures): Network error: Failed to ping master at 127.27.115.60:45299: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:51:00.650388 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:51:00.651052 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:51:04.500125 29653 maintenance_manager.cc:419] P ba9cc5c3257b466c8087bf6fc15b7284: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 597 bytes on disk
I20260809 01:51:04.516533 29652 maintenance_manager.cc:643] P ba9cc5c3257b466c8087bf6fc15b7284: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"dirs.queue_time_us":1027,"dirs.run_cpu_time_us":319,"dirs.run_wall_time_us":320,"thread_start_us":1172,"threads_started":2,"wal-append.queue_time_us":1159}
I20260809 01:51:05.072705 31036 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9b0e02ad573e476983ee8c2d623d2501)
I20260809 01:51:05.073424 31036 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b0e02ad573e476983ee8c2d623d2501" member_type: VOTER last_known_addr { host: "127.27.115.62" port: 33531 } } peers { permanent_uuid: "ba9cc5c3257b466c8087bf6fc15b7284" member_type: VOTER last_known_addr { host: "127.27.115.61" port: 37213 } } peers { permanent_uuid: "118ef761cca1412eac9223a88e991ff0" member_type: VOTER last_known_addr { host: "127.27.115.60" port: 45299 } }
I20260809 01:51:05.076627 31036 leader_election.cc:290] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531), 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299)
W20260809 01:51:05.079674 29648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260809 01:51:05.083956 29651 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9b0e02ad573e476983ee8c2d623d2501 (127.27.115.62:33531): Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:51:05.084520 29648 leader_election.cc:336] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 118ef761cca1412eac9223a88e991ff0 (127.27.115.60:45299): Network error: Client connection negotiation failed: client connection to 127.27.115.60:45299: connect: Connection refused (error 111)
I20260809 01:51:05.084810 29648 leader_election.cc:304] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba9cc5c3257b466c8087bf6fc15b7284; no voters: 118ef761cca1412eac9223a88e991ff0, 9b0e02ad573e476983ee8c2d623d2501
I20260809 01:51:05.085623 31036 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ba9cc5c3257b466c8087bf6fc15b7284 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 01:51:05.655355 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:51:05.655951 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:51:10.660423 29611 master_runner.cc:187] Error getting master registration for 127.27.115.62:33531: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111),
I20260809 01:51:10.661566 29611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260809 01:51:11.920279 29275 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:51:11.921036 29275 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
W20260809 01:51:11.921352 29275 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
F20260809 01:51:11.936139 29298 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260809 01:51:11.937919 29302 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.27.115.62:33531,127.27.115.61:37213,127.27.115.60:45299, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.115.62:33531: connect: Connection refused (error 111)
*** Aborted at 1786240271 (unix time) try "date -d @1786240271" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006dcc) received by PID 28108 (TID 0x7f401abfb700) from PID 28108; stack trace: ***
@ 0x2b07a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b2c83 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
*** Check failure stack trace: ***
@ 0x7f402e514980 (unknown) at ??:0
@ 0x7f40296cee87 gsignal at ??:0
@ 0x7f40296d07f1 abort at ??:0
@ 0x2b6796 __interceptor_abort at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f402c766d74 google::LogMessage::SendToLog() at ??:0
@ 0x7f402c767920 google::LogMessage::Flush() at ??:0
@ 0x7f402c76ca5b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35f0f6 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38c2fe _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38c1b1 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38b8fb std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2b0660 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f402e5096db start_thread at ??:0
@ 0x7f40297b161f clone at ??:0