Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260928 02:04:08.989122 8890 test_util.cc:268] Using random seed: 2068188562
2026-09-28T02:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:09Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskGCZ99a/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskGCZ99a/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/security.properties
2026-09-28 02:04:16: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskGCZ99a/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskGCZ99a/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260928 02:04:24.173197 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.190:39911
--webserver_interface=127.8.174.190
--webserver_port=0
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260928 02:04:24.525341 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:24.525663 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:24.525770 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:24.533342 9132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:24.533465 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:24.533512 9132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:24.533586 9132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:24.543167 9132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.190:39911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.174.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:24.544697 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:24.546571 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:24.563342 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:24.563272 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:24.563301 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:24.563535 9132 server_base.cc:1034] running on GCE node
I20260928 02:04:24.564761 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:24.566252 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:24.567521 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790561064567453 us; error 38 us; skew 500 ppm
I20260928 02:04:24.567914 9132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:24.570465 9132 webserver.cc:460] Webserver started at http://127.8.174.190:39585/ using document root <none> and password file <none>
I20260928 02:04:24.571228 9132 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:24.571345 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:24.571658 9132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:24.574537 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/instance:
uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-3l00"
I20260928 02:04:24.575392 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal/instance:
uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-3l00"
I20260928 02:04:24.580734 9132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:24.584528 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:24.586041 9132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:04:24.586248 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-3l00"
I20260928 02:04:24.586509 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:24.603402 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:24.604302 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:24.604604 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:24.622732 9132 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.190:39911
I20260928 02:04:24.622790 9197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.190:39911 every 8 connection(s)
I20260928 02:04:24.624315 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:24.628625 9198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.631093 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 9132
I20260928 02:04:24.631392 8890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal/instance
I20260928 02:04:24.634728 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--webserver_interface=127.8.174.189
--webserver_port=0
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260928 02:04:24.637631 9198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:04:24.658044 9198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } has no permanent_uuid. Determining permanent_uuid...
W20260928 02:04:24.659533 9148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:24.661369 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:24.710822 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } attempt: 1
W20260928 02:04:24.712746 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:24.781076 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } attempt: 2
W20260928 02:04:24.783095 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:24.874403 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } attempt: 3
W20260928 02:04:24.876349 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:24.965937 9199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:24.966254 9199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:24.966454 9199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:24.974228 9199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:24.974390 9199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:24.974447 9199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:24.974500 9199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:24.983918 9199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.174.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:24.985232 9199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:24.987043 9199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:24.999107 9207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:24.999143 9208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:24.999202 9210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:25.000082 9199 server_base.cc:1034] running on GCE node
I20260928 02:04:25.000934 9199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:25.002493 9199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:25.003742 9199 hybrid_clock.cc:648] HybridClock initialized: now 1790561065003658 us; error 50 us; skew 500 ppm
I20260928 02:04:25.004089 9199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:25.006410 9199 webserver.cc:460] Webserver started at http://127.8.174.189:34795/ using document root <none> and password file <none>
I20260928 02:04:25.007097 9199 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:25.007246 9199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:25.007552 9199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:25.012317 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/instance:
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:25.013114 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal/instance:
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:25.018872 9199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:04:25.019616 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } attempt: 4
W20260928 02:04:25.021539 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:25.022694 9216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:25.024251 9199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:04:25.024408 9199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:25.024580 9199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:25.038400 9199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:25.039222 9199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:25.039476 9199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:25.057183 9199 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.189:36913
I20260928 02:04:25.057206 9267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.189:36913 every 8 connection(s)
I20260928 02:04:25.058615 9199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
I20260928 02:04:25.061650 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 9199
I20260928 02:04:25.061844 8890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal/instance
I20260928 02:04:25.063176 9268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:25.063236 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.188:44075
--webserver_interface=127.8.174.188
--webserver_port=0
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260928 02:04:25.071086 9268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:04:25.086555 9268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:04:25.097314 9268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } has no permanent_uuid. Determining permanent_uuid...
W20260928 02:04:25.098690 9217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:25.100394 9268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:25.149737 9268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 1
W20260928 02:04:25.151684 9268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:25.219929 9268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 2
W20260928 02:04:25.221773 9268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:25.313033 9268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 3
W20260928 02:04:25.315186 9268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:25.320803 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } attempt: 5
I20260928 02:04:25.328091 9198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } has no permanent_uuid. Determining permanent_uuid...
W20260928 02:04:25.329977 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:25.381278 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 1
W20260928 02:04:25.383199 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:25.399813 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:25.400101 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:25.400182 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:25.407553 9269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:25.407676 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:25.407727 9269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:25.407780 9269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:25.417183 9269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.188:44075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.174.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:25.418634 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:25.420473 9269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:25.432711 9277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:25.432760 9280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:25.432606 9278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:25.433696 9269 server_base.cc:1034] running on GCE node
I20260928 02:04:25.434582 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:25.436043 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:25.437270 9269 hybrid_clock.cc:648] HybridClock initialized: now 1790561065437189 us; error 57 us; skew 500 ppm
I20260928 02:04:25.437624 9269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:25.440021 9269 webserver.cc:460] Webserver started at http://127.8.174.188:34749/ using document root <none> and password file <none>
I20260928 02:04:25.440728 9269 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:25.440850 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:25.441148 9269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:25.443892 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/instance:
uuid: "7017d177043645b5afe32c476055db7e"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:25.444634 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal/instance:
uuid: "7017d177043645b5afe32c476055db7e"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:25.450019 9269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:04:25.451455 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 2
W20260928 02:04:25.453200 9198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:25.453718 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:25.455444 9269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:04:25.455623 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
uuid: "7017d177043645b5afe32c476055db7e"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:25.455821 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:25.458400 9268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 4
W20260928 02:04:25.460399 9268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:25.476188 9269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:25.477108 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:25.477384 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:25.495517 9269 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.188:44075
I20260928 02:04:25.495528 9337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.188:44075 every 8 connection(s)
I20260928 02:04:25.497012 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/info.pb
I20260928 02:04:25.500892 9338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:25.501348 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 9269
I20260928 02:04:25.501534 8890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal/instance
I20260928 02:04:25.509104 9338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:04:25.537565 9338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:04:25.545017 9338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:04:25.555359 9338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Bootstrap starting.
I20260928 02:04:25.559103 9338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:25.559456 9198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 3
I20260928 02:04:25.560388 9338 log.cc:824] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:25.563714 9338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: No bootstrap required, opened a new log
I20260928 02:04:25.570017 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Bootstrap starting.
I20260928 02:04:25.571834 9338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.572306 9338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:25.572535 9338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7017d177043645b5afe32c476055db7e, State: Initialized, Role: FOLLOWER
I20260928 02:04:25.573434 9338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.573961 9198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:25.574414 9344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.574776 9344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:25.575273 9198 log.cc:824] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:25.575382 9338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:25.578130 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: No bootstrap required, opened a new log
I20260928 02:04:25.588027 9198 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.588595 9198 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:25.588819 9198 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09e70c2f7354c4e888b5aa6f0175a22, State: Initialized, Role: FOLLOWER
I20260928 02:04:25.589684 9198 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
W20260928 02:04:25.596827 9359 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 02:04:25.596611 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
W20260928 02:04:25.597007 9359 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:04:25.597086 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:25.598209 9198 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:25.588500 9269 master_runner.cc:186] Error getting master registration for 127.8.174.190:39911: OK, instance_id { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" instance_seqno: 1790561064617515 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:04:25.599080 9269 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:04:25.607371 9152 tablet.cc:2369] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:25.651130 9132 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: OK, instance_id { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" instance_seqno: 1790561065052306 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:04:25.663942 9132 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:04:25.670105 9376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:04:25.670360 9376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:04:25.678033 9362 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:25.678395 9362 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.680254 9362 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:25.682328 9313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e" is_pre_election: true
I20260928 02:04:25.683048 9313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f09e70c2f7354c4e888b5aa6f0175a22 in term 0.
I20260928 02:04:25.683796 9147 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22; no voters:
I20260928 02:04:25.685410 9362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:25.685561 9362 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:25.685669 9362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:25.689100 9362 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.692179 9313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e"
I20260928 02:04:25.692432 9313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:25.695904 9313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f09e70c2f7354c4e888b5aa6f0175a22 in term 1.
I20260928 02:04:25.697439 9147 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22; no voters:
I20260928 02:04:25.702940 9362 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 1 election: Requested vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:25.704206 9362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:25.704959 9362 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 1 LEADER]: Becoming Leader. State: Replica: f09e70c2f7354c4e888b5aa6f0175a22, State: Running, Role: LEADER
I20260928 02:04:25.706221 9362 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.722071 9379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09e70c2f7354c4e888b5aa6f0175a22. Latest consensus state: current_term: 1 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.722513 9379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: LEADER
W20260928 02:04:25.729827 9292 tablet.cc:2369] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:25.737094 9381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:25.744683 9381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:25.754329 9313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Refusing update from remote peer f09e70c2f7354c4e888b5aa6f0175a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:25.755548 9379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:25.759594 9268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } attempt: 5
I20260928 02:04:25.774010 9268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap starting.
I20260928 02:04:25.779902 9268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:25.781316 9268 log.cc:824] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:25.806064 9268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: No bootstrap required, opened a new log
I20260928 02:04:25.830672 9344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09e70c2f7354c4e888b5aa6f0175a22. Latest consensus state: current_term: 1 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.831039 9344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:25.841089 9268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.842989 9268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:25.843209 9268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Initialized, Role: FOLLOWER
I20260928 02:04:25.845463 9268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:25.853055 9387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.855469 9268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:25.855537 9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640"
I20260928 02:04:25.855880 9243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640" is_pre_election: true
I20260928 02:04:25.856489 9241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:25.857316 9387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:25.859061 9344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.859364 9344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:25.861778 9241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 1 FOLLOWER]: Refusing update from remote peer f09e70c2f7354c4e888b5aa6f0175a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:25.862730 9379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:25.865249 9379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.865552 9379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:25.876695 9381 catalog_manager.cc:1347] Generated new cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:25.876824 9381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:25.899266 9381 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:25.901244 9381 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:25.911541 9387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09e70c2f7354c4e888b5aa6f0175a22. Latest consensus state: current_term: 1 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.911860 9387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:25.911959 9379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.912213 9379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:25.912235 9396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:25.912570 9396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:25.939512 9381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Generated new TSK 0
I20260928 02:04:25.940970 9381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:25.942567 9381 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:25.943888 9381 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:25.965695 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.174.129:0
--local_ip_for_outbound_sockets=127.8.174.129
--webserver_interface=127.8.174.129
--webserver_port=0
--tserver_master_addrs=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260928 02:04:25.978978 9408 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:25.979086 9408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: loading cluster ID for follower catalog manager: success
I20260928 02:04:25.992554 9408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: acquiring CA information for follower catalog manager: success
I20260928 02:04:26.007865 9408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:04:26.438552 9411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:26.438894 9411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:26.439020 9411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:26.446534 9411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:26.446771 9411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.174.129
I20260928 02:04:26.456248 9411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.174.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.174.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.174.129
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:26.457561 9411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:26.459455 9411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:26.472491 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:26.472759 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:26.473726 9419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:26.474201 9411 server_base.cc:1034] running on GCE node
I20260928 02:04:26.474876 9411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:26.475943 9411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:26.477185 9411 hybrid_clock.cc:648] HybridClock initialized: now 1790561066477109 us; error 42 us; skew 500 ppm
I20260928 02:04:26.477505 9411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:26.479974 9411 webserver.cc:460] Webserver started at http://127.8.174.129:38453/ using document root <none> and password file <none>
I20260928 02:04:26.480682 9411 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:26.480783 9411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:26.481079 9411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:26.483721 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data/instance:
uuid: "ebc8a50f334349b68dfbeb1ae9fc3635"
format_stamp: "Formatted at 2026-09-28 02:04:26 on dist-test-slave-3l00"
I20260928 02:04:26.484454 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal/instance:
uuid: "ebc8a50f334349b68dfbeb1ae9fc3635"
format_stamp: "Formatted at 2026-09-28 02:04:26 on dist-test-slave-3l00"
I20260928 02:04:26.489753 9411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:04:26.493449 9425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:26.495070 9411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:26.495241 9411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal
uuid: "ebc8a50f334349b68dfbeb1ae9fc3635"
format_stamp: "Formatted at 2026-09-28 02:04:26 on dist-test-slave-3l00"
I20260928 02:04:26.495421 9411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:26.525826 9411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:26.526834 9411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:26.527153 9411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:26.528168 9411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:26.529981 9411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:26.530079 9411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:26.530200 9411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:26.530261 9411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:26.560513 9411 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.129:44359
I20260928 02:04:26.560570 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.129:44359 every 8 connection(s)
I20260928 02:04:26.561828 9411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:26.566538 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 9411
I20260928 02:04:26.566735 8890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:26.584879 9538 heartbeater.cc:347] Connected to a master server at 127.8.174.188:44075
I20260928 02:04:26.585264 9538 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:26.585860 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.174.130:0
--local_ip_for_outbound_sockets=127.8.174.130
--webserver_interface=127.8.174.130
--webserver_port=0
--tserver_master_addrs=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260928 02:04:26.586400 9538 heartbeater.cc:511] Master 127.8.174.188:44075 requested a full tablet report, sending...
I20260928 02:04:26.588869 9303 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:26.590571 9540 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:26.590597 9539 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:26.590713 9540 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:26.590740 9539 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:26.591182 9540 heartbeater.cc:511] Master 127.8.174.189:36913 requested a full tablet report, sending...
I20260928 02:04:26.591250 9539 heartbeater.cc:511] Master 127.8.174.190:39911 requested a full tablet report, sending...
I20260928 02:04:26.592878 9163 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:26.592906 9233 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:26.594625 9163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.174.129:47035
I20260928 02:04:26.599633 9359 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:26.599748 9359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: loading cluster ID for follower catalog manager: success
I20260928 02:04:26.603603 9359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: acquiring CA information for follower catalog manager: success
I20260928 02:04:26.606196 9359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:04:26.928000 9548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:26.928340 9548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:26.928439 9548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:26.935820 9548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:26.936009 9548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.174.130
I20260928 02:04:26.945207 9548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.174.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.174.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.174.130
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:26.946622 9548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:26.948518 9548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:26.962177 9553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:26.962282 9556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:26.962414 9554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:26.963253 9548 server_base.cc:1034] running on GCE node
I20260928 02:04:26.963889 9548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:26.964950 9548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:26.966204 9548 hybrid_clock.cc:648] HybridClock initialized: now 1790561066966134 us; error 31 us; skew 500 ppm
I20260928 02:04:26.966586 9548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:26.969033 9548 webserver.cc:460] Webserver started at http://127.8.174.130:42701/ using document root <none> and password file <none>
I20260928 02:04:26.969748 9548 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:26.969864 9548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:26.970158 9548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:26.972942 9548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data/instance:
uuid: "53ad00df981d4266a118559f648b1abe"
format_stamp: "Formatted at 2026-09-28 02:04:26 on dist-test-slave-3l00"
I20260928 02:04:26.973742 9548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal/instance:
uuid: "53ad00df981d4266a118559f648b1abe"
format_stamp: "Formatted at 2026-09-28 02:04:26 on dist-test-slave-3l00"
I20260928 02:04:26.979351 9548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:04:26.983222 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:26.984745 9548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:04:26.984889 9548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal
uuid: "53ad00df981d4266a118559f648b1abe"
format_stamp: "Formatted at 2026-09-28 02:04:26 on dist-test-slave-3l00"
I20260928 02:04:26.985067 9548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:27.013312 9548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:27.014365 9548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:27.014675 9548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:27.015647 9548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:27.017328 9548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:27.017407 9548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.017544 9548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:27.017596 9548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.047137 9548 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.130:39843
I20260928 02:04:27.047180 9674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.130:39843 every 8 connection(s)
I20260928 02:04:27.048450 9548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:27.054855 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 9548
I20260928 02:04:27.055083 8890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:27.062644 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.174.131:0
--local_ip_for_outbound_sockets=127.8.174.131
--webserver_interface=127.8.174.131
--webserver_port=0
--tserver_master_addrs=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260928 02:04:27.081283 9677 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:27.081708 9677 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:27.081308 9675 heartbeater.cc:347] Connected to a master server at 127.8.174.188:44075
I20260928 02:04:27.081990 9675 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:27.082402 9676 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:27.082542 9676 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:27.083010 9676 heartbeater.cc:511] Master 127.8.174.190:39911 requested a full tablet report, sending...
I20260928 02:04:27.083087 9675 heartbeater.cc:511] Master 127.8.174.188:44075 requested a full tablet report, sending...
I20260928 02:04:27.083113 9677 heartbeater.cc:511] Master 127.8.174.189:36913 requested a full tablet report, sending...
I20260928 02:04:27.084656 9233 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:27.084709 9163 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:27.085661 9303 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:27.085929 9163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.174.130:49637
I20260928 02:04:27.097481 9539 heartbeater.cc:503] Master 127.8.174.190:39911 was elected leader, sending a full tablet report...
I20260928 02:04:27.335667 9683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f09e70c2f7354c4e888b5aa6f0175a22)
I20260928 02:04:27.335876 9683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:27.337178 9683 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
I20260928 02:04:27.337950 9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7017d177043645b5afe32c476055db7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640" is_pre_election: true
I20260928 02:04:27.337854 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7017d177043645b5afe32c476055db7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:27.338795 9290 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:27.339253 9683 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 02:04:27.401574 9682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:27.401871 9682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:27.401968 9682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:27.409531 9682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:27.409746 9682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.174.131
I20260928 02:04:27.419104 9682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.174.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.174.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.174.131
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:27.420377 9682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:27.422173 9682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:27.435104 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:27.435070 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:27.435142 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:27.440053 9682 server_base.cc:1034] running on GCE node
I20260928 02:04:27.440730 9682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:27.441676 9682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:27.442945 9682 hybrid_clock.cc:648] HybridClock initialized: now 1790561067442848 us; error 55 us; skew 500 ppm
I20260928 02:04:27.443325 9682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:27.450034 9682 webserver.cc:460] Webserver started at http://127.8.174.131:35199/ using document root <none> and password file <none>
I20260928 02:04:27.450881 9682 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:27.451004 9682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:27.451315 9682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:27.454136 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data/instance:
uuid: "8c51770d4a61463c9330530e0d019d98"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-3l00"
I20260928 02:04:27.454962 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal/instance:
uuid: "8c51770d4a61463c9330530e0d019d98"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-3l00"
I20260928 02:04:27.460448 9682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:27.464072 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.465649 9682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:04:27.465837 9682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal
uuid: "8c51770d4a61463c9330530e0d019d98"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-3l00"
I20260928 02:04:27.466027 9682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:27.479959 9682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:27.480816 9682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:27.481104 9682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:27.482095 9682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:27.483753 9682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:27.483827 9682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.483948 9682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:27.483997 9682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.522046 9682 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.131:37953
I20260928 02:04:27.522080 9809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.131:37953 every 8 connection(s)
I20260928 02:04:27.523346 9682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:27.534246 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 9682
I20260928 02:04:27.534431 8890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:27.541541 9810 heartbeater.cc:347] Connected to a master server at 127.8.174.188:44075
I20260928 02:04:27.541571 9812 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:27.541604 9811 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:27.541911 9811 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:27.541908 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:27.541905 9810 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:27.543005 9810 heartbeater.cc:511] Master 127.8.174.188:44075 requested a full tablet report, sending...
I20260928 02:04:27.543040 9811 heartbeater.cc:511] Master 127.8.174.190:39911 requested a full tablet report, sending...
I20260928 02:04:27.543835 9812 heartbeater.cc:511] Master 127.8.174.189:36913 requested a full tablet report, sending...
I20260928 02:04:27.544782 9233 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:27.544782 9303 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:27.544870 9163 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:27.545864 9163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.174.131:37339
I20260928 02:04:27.561640 8890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:04:27.588786 9676 heartbeater.cc:503] Master 127.8.174.190:39911 was elected leader, sending a full tablet report...
I20260928 02:04:27.591203 9163 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.591436 9161 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_5e198fc5d5f94fca9b5d9341684b8187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.591150 9162 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_b2462a042ad34570ac6321c1d9e4212c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.592459 9160 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_ccd4aeece7af4142a6e4c84a9bc39b89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.592556 9159 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:27.600212 9161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5e198fc5d5f94fca9b5d9341684b8187 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:27.604833 9157 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.604787 9158 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_657e2574ee3d4050ab7aade380a35acc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.608042 9156 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:27.608603 9163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ae2bdbd94c92423aaea96f97b87e34ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:27.612068 9155 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_483ce76de8ea43dea7e133f72fa81aa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.616204 9154 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48950:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.637594 8890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu pid 9132
I20260928 02:04:27.639050 9193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:27.737898 9857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f09e70c2f7354c4e888b5aa6f0175a22)
I20260928 02:04:27.738194 9857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:27.739768 9857 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:27.740530 9313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e" is_pre_election: true
I20260928 02:04:27.791860 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:27.793131 8890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu with pid 9132
I20260928 02:04:27.793051 9220 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640; no voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:27.793813 9857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 02:04:27.820336 9820 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.190:39911)
W20260928 02:04:27.829138 9823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:27.822136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.174.190:39911 (local address 127.0.0.1:48962)
0928 02:04:27.824309 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:27.824333 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:27.824510 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:27.827446 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1886}
I20260928 02:04:27.911634 9858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f09e70c2f7354c4e888b5aa6f0175a22)
I20260928 02:04:27.911810 9858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:27.912474 9858 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
I20260928 02:04:27.912931 9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7017d177043645b5afe32c476055db7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640" is_pre_election: true
I20260928 02:04:27.913261 9241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7017d177043645b5afe32c476055db7e in term 1.
I20260928 02:04:27.913832 9288 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e; no voters:
I20260928 02:04:27.915328 9858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:04:27.915522 9858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f09e70c2f7354c4e888b5aa6f0175a22)
I20260928 02:04:27.915724 9858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 1 FOLLOWER]: Advancing to term 2
W20260928 02:04:27.915827 9290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:27.919199 9858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:27.923837 9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7017d177043645b5afe32c476055db7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640"
I20260928 02:04:27.924117 9241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:04:27.927199 9241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7017d177043645b5afe32c476055db7e in term 2.
W20260928 02:04:27.927521 9290 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:27.927762 9288 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e; no voters:
W20260928 02:04:27.929198 9290 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:27.934501 9858 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 2 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
I20260928 02:04:27.934770 9858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:04:27.934928 9858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 2 LEADER]: Becoming Leader. State: Replica: 7017d177043645b5afe32c476055db7e, State: Running, Role: LEADER
I20260928 02:04:27.935385 9858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:27.941542 9858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7017d177043645b5afe32c476055db7e. Latest consensus state: current_term: 2 leader_uuid: "7017d177043645b5afe32c476055db7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:27.941833 9858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:27.942625 9863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:27.945140 9863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:27.946352 9863 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:27.946527 9863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:27.948290 9863 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:27.949512 9863 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Loaded TSK: 0
I20260928 02:04:27.951141 9863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:27.952347 9863 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:27.953477 9863 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:27.959779 9300 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.960073 9298 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.960173 9299 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_b2462a042ad34570ac6321c1d9e4212c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.960855 9296 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_ccd4aeece7af4142a6e4c84a9bc39b89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.960930 9297 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_5e198fc5d5f94fca9b5d9341684b8187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.961586 9294 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_657e2574ee3d4050ab7aade380a35acc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.961639 9295 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.962807 9302 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_483ce76de8ea43dea7e133f72fa81aa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.959738 9301 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:27.963994 9303 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:27.971227 9299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b2462a042ad34570ac6321c1d9e4212c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:27.973101 9297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5e198fc5d5f94fca9b5d9341684b8187 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:27.974016 9296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:28.048002 9811 heartbeater.cc:503] Master 127.8.174.190:39911 was elected leader, sending a full tablet report...
W20260928 02:04:28.050212 9811 heartbeater.cc:644] Failed to heartbeat to 127.8.174.190:39911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:28.093358 9676 heartbeater.cc:644] Failed to heartbeat to 127.8.174.190:39911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:28.109985 9539 heartbeater.cc:644] Failed to heartbeat to 127.8.174.190:39911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:28.136196 9299 catalog_manager.cc:2148] created HMS catalog entry for table [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:28.141638 9241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 2 FOLLOWER]: Refusing update from remote peer 7017d177043645b5afe32c476055db7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260928 02:04:28.142535 9861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260928 02:04:28.143433 9290 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e -> Peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Couldn't send request to peer f09e70c2f7354c4e888b5aa6f0175a22. Status: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:04:28.145910 9857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7017d177043645b5afe32c476055db7e. Latest consensus state: current_term: 2 leader_uuid: "7017d177043645b5afe32c476055db7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:28.146203 9857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:28.147969 9858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7017d177043645b5afe32c476055db7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:28.148263 9858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:28.148849 9868 mvcc.cc:205] Tried to move back new op lower bound from 7334138135096139776 to 7334138134266966016. Current Snapshot: MvccSnapshot[applied={T|T < 7334138135096139776}]
I20260928 02:04:28.155048 9299 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49590:
table { table_name: "default.Table_b2462a042ad34570ac6321c1d9e4212c" } new_table_name: "default.Table_b6652c20a8c44c2d8e1ec6246b2159a9" modify_external_catalogs: true
I20260928 02:04:28.156002 9857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7017d177043645b5afe32c476055db7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:28.156297 9857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:28.196921 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccfcbc6f0a3489dae7c315c019ed94f (DEFAULT_TABLE table=default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.198607 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccfcbc6f0a3489dae7c315c019ed94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.224908 9610 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccfcbc6f0a3489dae7c315c019ed94f (DEFAULT_TABLE table=default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.226902 9610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccfcbc6f0a3489dae7c315c019ed94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.228097 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccfcbc6f0a3489dae7c315c019ed94f (DEFAULT_TABLE table=default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.229903 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccfcbc6f0a3489dae7c315c019ed94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.233034 9877 tablet_bootstrap.cc:492] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Bootstrap starting.
I20260928 02:04:28.237046 9877 tablet_bootstrap.cc:654] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.239557 9877 log.cc:824] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:28.249347 9877 tablet_bootstrap.cc:492] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: No bootstrap required, opened a new log
I20260928 02:04:28.249764 9877 ts_tablet_manager.cc:1397] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260928 02:04:28.250684 9880 tablet_bootstrap.cc:492] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Bootstrap starting.
I20260928 02:04:28.256382 9880 tablet_bootstrap.cc:654] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.257426 9879 tablet_bootstrap.cc:492] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Bootstrap starting.
I20260928 02:04:28.260071 9877 raft_consensus.cc:348] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } }
I20260928 02:04:28.260619 9877 raft_consensus.cc:374] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.260795 9877 raft_consensus.cc:729] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.261638 9877 consensus_queue.cc:260] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } }
I20260928 02:04:28.261798 9880 log.cc:824] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:28.264554 9879 tablet_bootstrap.cc:654] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.266183 9879 log.cc:824] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:28.272328 9879 tablet_bootstrap.cc:492] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: No bootstrap required, opened a new log
I20260928 02:04:28.272713 9879 ts_tablet_manager.cc:1397] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20260928 02:04:28.277694 9880 tablet_bootstrap.cc:492] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: No bootstrap required, opened a new log
I20260928 02:04:28.278158 9880 ts_tablet_manager.cc:1397] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20260928 02:04:28.288774 9538 heartbeater.cc:503] Master 127.8.174.188:44075 was elected leader, sending a full tablet report...
I20260928 02:04:28.290166 9880 raft_consensus.cc:348] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } }
I20260928 02:04:28.290719 9880 raft_consensus.cc:374] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.290832 9880 raft_consensus.cc:729] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51770d4a61463c9330530e0d019d98, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.291553 9880 consensus_queue.cc:260] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } }
I20260928 02:04:28.288771 9877 ts_tablet_manager.cc:1428] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent starting tablet: real 0.039s user 0.019s sys 0.012s
I20260928 02:04:28.299866 9810 heartbeater.cc:503] Master 127.8.174.188:44075 was elected leader, sending a full tablet report...
I20260928 02:04:28.300709 9880 ts_tablet_manager.cc:1428] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Time spent starting tablet: real 0.022s user 0.013s sys 0.007s
I20260928 02:04:28.308511 9879 raft_consensus.cc:348] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } }
I20260928 02:04:28.309993 9879 raft_consensus.cc:374] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.310107 9879 raft_consensus.cc:729] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ad00df981d4266a118559f648b1abe, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.311952 9879 consensus_queue.cc:260] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } }
I20260928 02:04:28.314276 9675 heartbeater.cc:503] Master 127.8.174.188:44075 was elected leader, sending a full tablet report...
I20260928 02:04:28.315083 9879 ts_tablet_manager.cc:1428] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Time spent starting tablet: real 0.042s user 0.014s sys 0.003s
W20260928 02:04:28.315157 9541 tablet.cc:2369] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:04:28.360 [HMS - ERROR - pool-6-thread-1] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_5e198fc5d5f94fca9b5d9341684b8187 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
W20260928 02:04:28.363970 9297 catalog_manager.cc:2144] Already present: failed to create HMS catalog entry for table [id=d078b77f9f504805aac87a457a893aab]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_5e198fc5d5f94fca9b5d9341684b8187 already exists)
I20260928 02:04:28.374347 9297 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ccfcbc6f0a3489dae7c315c019ed94f from {username='slave'} at 127.0.0.1:49590
I20260928 02:04:28.383761 9297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Sending DeleteTablet for 3 replicas of tablet 4ccfcbc6f0a3489dae7c315c019ed94f
I20260928 02:04:28.385368 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ccfcbc6f0a3489dae7c315c019ed94f with delete_type TABLET_DATA_DELETED (replaced by tablet 006d1c27cc874f11bd66fb947942a96a) from {username='slave'} at 127.0.0.1:46782
I20260928 02:04:28.386127 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ccfcbc6f0a3489dae7c315c019ed94f with delete_type TABLET_DATA_DELETED (replaced by tablet 006d1c27cc874f11bd66fb947942a96a) from {username='slave'} at 127.0.0.1:55792
I20260928 02:04:28.388089 9887 tablet_replica.cc:330] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: stopping tablet replica
I20260928 02:04:28.388759 9887 raft_consensus.cc:2229] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:28.389362 9887 raft_consensus.cc:2258] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:28.391353 9887 ts_tablet_manager.cc:1905] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:28.393975 9610 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ccfcbc6f0a3489dae7c315c019ed94f with delete_type TABLET_DATA_DELETED (replaced by tablet 006d1c27cc874f11bd66fb947942a96a) from {username='slave'} at 127.0.0.1:57236
I20260928 02:04:28.395316 9297 catalog_manager.cc:6565] ReplaceTablet: tablet 4ccfcbc6f0a3489dae7c315c019ed94f deleted and replaced by tablet 006d1c27cc874f11bd66fb947942a96a
I20260928 02:04:28.400214 9887 ts_tablet_manager.cc:1918] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260928 02:04:28.400503 9887 log.cc:1196] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Deleting WAL directory at /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal/wals/4ccfcbc6f0a3489dae7c315c019ed94f
I20260928 02:04:28.401103 9887 ts_tablet_manager.cc:1939] T 4ccfcbc6f0a3489dae7c315c019ed94f P 8c51770d4a61463c9330530e0d019d98: Deleting consensus metadata
I20260928 02:04:28.403249 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 006d1c27cc874f11bd66fb947942a96a (DEFAULT_TABLE table=default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.403599 9290 catalog_manager.cc:4841] TS 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): tablet 4ccfcbc6f0a3489dae7c315c019ed94f (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]) successfully deleted
I20260928 02:04:28.403609 9609 tablet_service.cc:1467] Processing CreateTablet for tablet 006d1c27cc874f11bd66fb947942a96a (DEFAULT_TABLE table=default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.403852 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006d1c27cc874f11bd66fb947942a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.404122 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006d1c27cc874f11bd66fb947942a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.406646 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 006d1c27cc874f11bd66fb947942a96a (DEFAULT_TABLE table=default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.407061 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006d1c27cc874f11bd66fb947942a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.407120 9297 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ccfcbc6f0a3489dae7c315c019ed94f from {username='slave'} at 127.0.0.1:49590
I20260928 02:04:28.408854 9297 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ccfcbc6f0a3489dae7c315c019ed94f from {username='slave'} at 127.0.0.1:49590
I20260928 02:04:28.410380 9297 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ccfcbc6f0a3489dae7c315c019ed94f from {username='slave'} at 127.0.0.1:49590
I20260928 02:04:28.411645 9877 tablet_bootstrap.cc:492] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635: Bootstrap starting.
I20260928 02:04:28.411729 9297 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ccfcbc6f0a3489dae7c315c019ed94f from {username='slave'} at 127.0.0.1:49590
I20260928 02:04:28.412804 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.190:39911
--webserver_interface=127.8.174.190
--webserver_port=39585
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260928 02:04:28.413384 9877 tablet_bootstrap.cc:654] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.418027 9296 catalog_manager.cc:2148] created HMS catalog entry for table [id=972b59b79c6b471a82a92801303d3898]
W20260928 02:04:28.421854 9295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cb325c0bff7b4d7ca343532d7e14b25f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:28.422163 9300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_21e7ecfcfe254c049a51ef89351c6a38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:28.422190 9303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_406b409ab888471581f1119b16f99179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:28.421845 9294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_657e2574ee3d4050ab7aade380a35acc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:28.423029 9298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ae2bdbd94c92423aaea96f97b87e34ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:28.423070 9301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8d14d0f90c2b4abea725b02c59da2573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 02:04:28.421856 9302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_483ce76de8ea43dea7e133f72fa81aa1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:28.424962 9297 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_7f2d941f68d647be9c91a40928667e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:28.425259 9296 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49590:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:28.428197 9886 tablet_replica.cc:330] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: stopping tablet replica
I20260928 02:04:28.429021 9886 raft_consensus.cc:2229] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:28.429574 9886 raft_consensus.cc:2258] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Raft consensus is shut down!
F20260928 02:04:28.430732 9358 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
I20260928 02:04:28.436591 9473 tablet_service.cc:1467] Processing CreateTablet for tablet e65189e85b2641098a1bdfca68c049c4 (DEFAULT_TABLE table=default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.437510 9879 tablet_bootstrap.cc:492] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe: Bootstrap starting.
@ 0x7f7996d279ad google::LogMessage::Fail() at ??:0
I20260928 02:04:28.438182 9608 tablet_service.cc:1467] Processing CreateTablet for tablet e65189e85b2641098a1bdfca68c049c4 (DEFAULT_TABLE table=default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.438812 9608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65189e85b2641098a1bdfca68c049c4. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7996d2a88a google::LogMessage::SendToLog() at ??:0
I20260928 02:04:28.439299 9879 tablet_bootstrap.cc:654] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe: Neither blocks nor log segments found. Creating new log.
@ 0x7f7996d274de google::LogMessage::Flush() at ??:0
I20260928 02:04:28.443589 9744 tablet_service.cc:1467] Processing CreateTablet for tablet e65189e85b2641098a1bdfca68c049c4 (DEFAULT_TABLE table=default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.444098 9744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65189e85b2641098a1bdfca68c049c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.449954 9880 tablet_bootstrap.cc:492] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98: Bootstrap starting.
I20260928 02:04:28.450518 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65189e85b2641098a1bdfca68c049c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.451591 9880 tablet_bootstrap.cc:654] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98: Neither blocks nor log segments found. Creating new log.
@ 0x7f7996d276d1 google::LogMessage::~LogMessage() at ??:0
I20260928 02:04:28.461186 9886 ts_tablet_manager.cc:1905] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:28.465235 9888 tablet_replica.cc:330] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: stopping tablet replica
I20260928 02:04:28.465812 9888 raft_consensus.cc:2229] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:28.466352 9888 raft_consensus.cc:2258] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:28.472777 9888 ts_tablet_manager.cc:1905] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:28.475296 9877 tablet_bootstrap.cc:492] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635: No bootstrap required, opened a new log
I20260928 02:04:28.475495 9877 ts_tablet_manager.cc:1397] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.006s
I20260928 02:04:28.476032 9879 tablet_bootstrap.cc:492] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe: No bootstrap required, opened a new log
I20260928 02:04:28.476249 9879 ts_tablet_manager.cc:1397] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.005s
I20260928 02:04:28.476755 9877 raft_consensus.cc:348] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.476979 9877 raft_consensus.cc:374] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.477070 9877 raft_consensus.cc:729] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.477238 9879 raft_consensus.cc:348] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.477489 9877 consensus_queue.cc:260] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.477777 9879 raft_consensus.cc:374] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.477828 9886 ts_tablet_manager.cc:1918] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260928 02:04:28.477880 9879 raft_consensus.cc:729] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ad00df981d4266a118559f648b1abe, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.477983 9886 log.cc:1196] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Deleting WAL directory at /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal/wals/4ccfcbc6f0a3489dae7c315c019ed94f
I20260928 02:04:28.478225 9879 consensus_queue.cc:260] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.478478 9886 ts_tablet_manager.cc:1939] T 4ccfcbc6f0a3489dae7c315c019ed94f P ebc8a50f334349b68dfbeb1ae9fc3635: Deleting consensus metadata
I20260928 02:04:28.478523 9880 tablet_bootstrap.cc:492] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98: No bootstrap required, opened a new log
I20260928 02:04:28.478678 9880 ts_tablet_manager.cc:1397] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.006s
I20260928 02:04:28.479460 9880 raft_consensus.cc:348] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.479657 9880 raft_consensus.cc:374] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.479755 9880 raft_consensus.cc:729] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51770d4a61463c9330530e0d019d98, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.480993 9879 ts_tablet_manager.cc:1428] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
@ 0x7f79ac8c9ba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260928 02:04:28.483340 9879 tablet_bootstrap.cc:492] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe: Bootstrap starting.
I20260928 02:04:28.485045 9879 tablet_bootstrap.cc:654] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.485159 9880 consensus_queue.cc:260] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.489893 9880 ts_tablet_manager.cc:1428] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
@ 0x7f79ac8c8901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260928 02:04:28.493301 9880 tablet_bootstrap.cc:492] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: Bootstrap starting.
I20260928 02:04:28.495553 9289 catalog_manager.cc:4841] TS ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): tablet 4ccfcbc6f0a3489dae7c315c019ed94f (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]) successfully deleted
I20260928 02:04:28.498181 9877 ts_tablet_manager.cc:1428] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20260928 02:04:28.501327 9877 tablet_bootstrap.cc:492] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: Bootstrap starting.
I20260928 02:04:28.509104 9888 ts_tablet_manager.cc:1918] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260928 02:04:28.509280 9888 log.cc:1196] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Deleting WAL directory at /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-1/wal/wals/4ccfcbc6f0a3489dae7c315c019ed94f
I20260928 02:04:28.509675 9888 ts_tablet_manager.cc:1939] T 4ccfcbc6f0a3489dae7c315c019ed94f P 53ad00df981d4266a118559f648b1abe: Deleting consensus metadata
I20260928 02:04:28.511241 9880 tablet_bootstrap.cc:654] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: Neither blocks nor log segments found. Creating new log.
@ 0x7f79ac8cf66c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260928 02:04:28.513115 9877 tablet_bootstrap.cc:654] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.517297 9879 tablet_bootstrap.cc:492] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe: No bootstrap required, opened a new log
I20260928 02:04:28.517477 9879 ts_tablet_manager.cc:1397] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20260928 02:04:28.518261 9879 raft_consensus.cc:348] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.518538 9879 raft_consensus.cc:374] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.518611 9879 raft_consensus.cc:729] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ad00df981d4266a118559f648b1abe, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.518906 9879 consensus_queue.cc:260] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.519779 9879 ts_tablet_manager.cc:1428] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
@ 0x7f79ac8cf4e9 std::_Function_handler<>::_M_invoke() at ??:0
W20260928 02:04:28.531764 9813 tablet.cc:2369] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260928 02:04:28.519387 9880 tablet_bootstrap.cc:492] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: No bootstrap required, opened a new log
I20260928 02:04:28.538879 9880 ts_tablet_manager.cc:1397] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.002s
I20260928 02:04:28.539825 9880 raft_consensus.cc:348] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.540073 9880 raft_consensus.cc:374] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.540139 9880 raft_consensus.cc:729] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51770d4a61463c9330530e0d019d98, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.540710 9880 consensus_queue.cc:260] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.541352 9295 catalog_manager.cc:2148] created HMS catalog entry for table [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:28.545499 9884 raft_consensus.cc:482] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:28.545781 9884 raft_consensus.cc:504] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.547827 9884 leader_election.cc:288] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
@ 0x7f7999951eaf kudu::Thread::SuperviseThread() at ??:0
I20260928 02:04:28.555917 9295 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49590:
table { table_name: "default.Table_ccd4aeece7af4142a6e4c84a9bc39b89" } new_table_name: "default.Table_e5b9b953426e458a8839ebcc1ea8436f" modify_external_catalogs: true
W20260928 02:04:28.560400 9678 tablet.cc:2369] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:28.562667 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d1c27cc874f11bd66fb947942a96a" candidate_uuid: "8c51770d4a61463c9330530e0d019d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe" is_pre_election: true
I20260928 02:04:28.563272 9630 raft_consensus.cc:2454] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c51770d4a61463c9330530e0d019d98 in term 0.
I20260928 02:04:28.565757 9698 leader_election.cc:302] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98; no voters:
I20260928 02:04:28.572378 9610 tablet_service.cc:1467] Processing CreateTablet for tablet e02df1dfaba348449be5816040ea63bd (DEFAULT_TABLE table=default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.572744 9744 tablet_service.cc:1467] Processing CreateTablet for tablet e02df1dfaba348449be5816040ea63bd (DEFAULT_TABLE table=default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.572861 9610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02df1dfaba348449be5816040ea63bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.573139 9744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02df1dfaba348449be5816040ea63bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.573915 9884 raft_consensus.cc:2781] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:28.574084 9884 raft_consensus.cc:482] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:28.574184 9884 raft_consensus.cc:3037] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.577248 9884 raft_consensus.cc:504] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.578593 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d1c27cc874f11bd66fb947942a96a" candidate_uuid: "8c51770d4a61463c9330530e0d019d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe"
I20260928 02:04:28.578697 9473 tablet_service.cc:1467] Processing CreateTablet for tablet e02df1dfaba348449be5816040ea63bd (DEFAULT_TABLE table=default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:28.578821 9630 raft_consensus.cc:3037] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.579157 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02df1dfaba348449be5816040ea63bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:28.581759 9630 raft_consensus.cc:2454] T 006d1c27cc874f11bd66fb947942a96a P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c51770d4a61463c9330530e0d019d98 in term 1.
I20260928 02:04:28.582356 9698 leader_election.cc:302] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98; no voters:
I20260928 02:04:28.584211 9884 leader_election.cc:288] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 1 election: Requested vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:28.584277 9877 tablet_bootstrap.cc:492] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: No bootstrap required, opened a new log
I20260928 02:04:28.584424 9884 raft_consensus.cc:2781] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:28.584444 9877 ts_tablet_manager.cc:1397] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.002s
I20260928 02:04:28.584553 9884 raft_consensus.cc:686] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [term 1 LEADER]: Becoming Leader. State: Replica: 8c51770d4a61463c9330530e0d019d98, State: Running, Role: LEADER
I20260928 02:04:28.584959 9884 consensus_queue.cc:237] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.585248 9877 raft_consensus.cc:348] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.585480 9877 raft_consensus.cc:374] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.585559 9877 raft_consensus.cc:729] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.586243 9879 tablet_bootstrap.cc:492] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe: Bootstrap starting.
I20260928 02:04:28.587268 9880 ts_tablet_manager.cc:1428] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20260928 02:04:28.588061 9879 tablet_bootstrap.cc:654] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.589458 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d1c27cc874f11bd66fb947942a96a" candidate_uuid: "8c51770d4a61463c9330530e0d019d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" is_pre_election: true
I20260928 02:04:28.590013 9493 raft_consensus.cc:2454] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c51770d4a61463c9330530e0d019d98 in term 0.
I20260928 02:04:28.590893 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d1c27cc874f11bd66fb947942a96a" candidate_uuid: "8c51770d4a61463c9330530e0d019d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635"
I20260928 02:04:28.591063 9493 raft_consensus.cc:3037] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.594338 9493 raft_consensus.cc:2454] T 006d1c27cc874f11bd66fb947942a96a P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c51770d4a61463c9330530e0d019d98 in term 1.
I20260928 02:04:28.587752 9877 consensus_queue.cc:260] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
@ 0x7f799cc2b6db start_thread at ??:0
@ 0x7f7995bc161f clone at ??:0
I20260928 02:04:28.602627 9877 ts_tablet_manager.cc:1428] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20260928 02:04:28.604040 9880 tablet_bootstrap.cc:492] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98: Bootstrap starting.
I20260928 02:04:28.608903 9877 tablet_bootstrap.cc:492] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635: Bootstrap starting.
I20260928 02:04:28.611869 9879 tablet_bootstrap.cc:492] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe: No bootstrap required, opened a new log
I20260928 02:04:28.612037 9879 ts_tablet_manager.cc:1397] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.007s
I20260928 02:04:28.612859 9879 raft_consensus.cc:348] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.613112 9879 raft_consensus.cc:374] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.613198 9879 raft_consensus.cc:729] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ad00df981d4266a118559f648b1abe, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.613462 9879 consensus_queue.cc:260] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.617089 9879 ts_tablet_manager.cc:1428] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260928 02:04:28.619340 9880 tablet_bootstrap.cc:654] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.623165 9877 tablet_bootstrap.cc:654] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:28.644876 9877 tablet_bootstrap.cc:492] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635: No bootstrap required, opened a new log
I20260928 02:04:28.644877 9880 tablet_bootstrap.cc:492] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98: No bootstrap required, opened a new log
I20260928 02:04:28.645040 9877 ts_tablet_manager.cc:1397] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.002s
I20260928 02:04:28.645041 9880 ts_tablet_manager.cc:1397] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.005s
I20260928 02:04:28.645824 9880 raft_consensus.cc:348] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.645875 9877 raft_consensus.cc:348] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.646100 9880 raft_consensus.cc:374] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.646108 9877 raft_consensus.cc:374] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:28.646185 9877 raft_consensus.cc:729] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.646184 9880 raft_consensus.cc:729] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51770d4a61463c9330530e0d019d98, State: Initialized, Role: FOLLOWER
I20260928 02:04:28.646471 9877 consensus_queue.cc:260] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.646507 9880 consensus_queue.cc:260] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.647032 9877 ts_tablet_manager.cc:1428] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:04:28.651095 9880 ts_tablet_manager.cc:1428] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260928 02:04:28.668020 9881 raft_consensus.cc:482] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:28.668273 9881 raft_consensus.cc:504] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.670364 9881 leader_election.cc:288] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:28.679934 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02df1dfaba348449be5816040ea63bd" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe" is_pre_election: true
I20260928 02:04:28.680248 9630 raft_consensus.cc:2454] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 0.
I20260928 02:04:28.680845 9426 leader_election.cc:302] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ad00df981d4266a118559f648b1abe, ebc8a50f334349b68dfbeb1ae9fc3635; no voters:
I20260928 02:04:28.681365 9881 raft_consensus.cc:2781] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:28.681478 9881 raft_consensus.cc:482] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:28.681567 9881 raft_consensus.cc:3037] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.687898 9881 raft_consensus.cc:504] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.689522 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02df1dfaba348449be5816040ea63bd" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe"
I20260928 02:04:28.689796 9630 raft_consensus.cc:3037] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.692628 9630 raft_consensus.cc:2454] T e02df1dfaba348449be5816040ea63bd P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 1.
I20260928 02:04:28.693116 9426 leader_election.cc:302] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ad00df981d4266a118559f648b1abe, ebc8a50f334349b68dfbeb1ae9fc3635; no voters:
I20260928 02:04:28.693867 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02df1dfaba348449be5816040ea63bd" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c51770d4a61463c9330530e0d019d98" is_pre_election: true
I20260928 02:04:28.694339 9765 raft_consensus.cc:2454] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 0.
I20260928 02:04:28.702765 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02df1dfaba348449be5816040ea63bd" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c51770d4a61463c9330530e0d019d98"
I20260928 02:04:28.703027 9765 raft_consensus.cc:3037] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.706176 9765 raft_consensus.cc:2454] T e02df1dfaba348449be5816040ea63bd P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 1.
I20260928 02:04:28.707376 9881 leader_election.cc:288] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 election: Requested vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:28.707585 9881 raft_consensus.cc:2781] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:28.707695 9881 raft_consensus.cc:686] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 LEADER]: Becoming Leader. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Running, Role: LEADER
I20260928 02:04:28.708047 9881 consensus_queue.cc:237] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.720549 9881 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:28.720767 9881 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.722018 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65189e85b2641098a1bdfca68c049c4" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c51770d4a61463c9330530e0d019d98" is_pre_election: true
I20260928 02:04:28.722280 9765 raft_consensus.cc:2454] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 0.
I20260928 02:04:28.722760 9429 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635; no voters:
I20260928 02:04:28.723112 9881 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:28.724939 9881 raft_consensus.cc:2781] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:28.725054 9881 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:28.725153 9881 raft_consensus.cc:3037] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.731367 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65189e85b2641098a1bdfca68c049c4" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe" is_pre_election: true
I20260928 02:04:28.731637 9630 raft_consensus.cc:2454] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 0.
I20260928 02:04:28.732522 9881 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.733618 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65189e85b2641098a1bdfca68c049c4" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe"
I20260928 02:04:28.733819 9630 raft_consensus.cc:3037] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.736613 9630 raft_consensus.cc:2454] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 1.
I20260928 02:04:28.737214 9426 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ad00df981d4266a118559f648b1abe, ebc8a50f334349b68dfbeb1ae9fc3635; no voters:
I20260928 02:04:28.737564 9912 raft_consensus.cc:2781] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:28.737746 9912 raft_consensus.cc:686] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 LEADER]: Becoming Leader. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Running, Role: LEADER
I20260928 02:04:28.738070 9912 consensus_queue.cc:237] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:28.740119 9881 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 election: Requested vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:28.741170 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65189e85b2641098a1bdfca68c049c4" candidate_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c51770d4a61463c9330530e0d019d98"
I20260928 02:04:28.741413 9765 raft_consensus.cc:3037] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:28.744052 9765 raft_consensus.cc:2454] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc8a50f334349b68dfbeb1ae9fc3635 in term 1.
W20260928 02:04:28.876266 9564 connection.cc:535] server connection from 127.0.0.1:57236 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:28.877288 9817 connection.cc:535] client connection to 127.8.174.188:44075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:28.878006 9817 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.188:44075)
W20260928 02:04:28.878692 9817 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.188:44075)
W20260928 02:04:28.879364 9817 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.188:44075)
W20260928 02:04:28.955612 9890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:28.955941 9890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:28.956017 9890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:28.964201 9890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:28.964304 9890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:28.964353 9890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:28.964421 9890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:28.974535 9890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.190:39911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.174.190
--webserver_port=39585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:28.975929 9890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:28.977784 9890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:28.990355 9927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:28.990399 9929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:28.990428 9926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:28.991425 9890 server_base.cc:1034] running on GCE node
I20260928 02:04:28.992213 9890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:28.993768 9890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:28.995002 9890 hybrid_clock.cc:648] HybridClock initialized: now 1790561068994931 us; error 41 us; skew 500 ppm
I20260928 02:04:28.995357 9890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:28.997680 9890 webserver.cc:460] Webserver started at http://127.8.174.190:39585/ using document root <none> and password file <none>
I20260928 02:04:28.998413 9890 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:28.998560 9890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:29.004902 9890 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:04:29.008172 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:29.009644 9890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:04:29.009806 9890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-3l00"
I20260928 02:04:29.010859 9890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:29.037621 9890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:29.038570 9890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:29.038857 9890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:29.063102 9890 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.190:39911
I20260928 02:04:29.063122 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.190:39911 every 8 connection(s)
I20260928 02:04:29.065150 9890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:29.075174 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 9890
I20260928 02:04:29.076684 9987 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:29.082844 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Bootstrap starting.
I20260928 02:04:29.095249 9988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7017d177043645b5afe32c476055db7e)
I20260928 02:04:29.095443 9987 log.cc:824] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:29.096256 9988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:29.097733 9988 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:29.109171 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:04:29.110810 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Bootstrap complete.
W20260928 02:04:29.114550 9217 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:29.123724 9987 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:29.125023 9987 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09e70c2f7354c4e888b5aa6f0175a22, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.125813 9987 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:29.126073 9676 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:29.126843 9997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:29.127252 9997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:29.128201 9987 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:29.128262 9962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:29.128886 9962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 1.
I20260928 02:04:29.129455 9220 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 7017d177043645b5afe32c476055db7e
I20260928 02:04:29.129732 9988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 02:04:29.129848 9988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7017d177043645b5afe32c476055db7e)
I20260928 02:04:29.129921 9988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:04:29.132023 9988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:29.132707 9988 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 3 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:29.133116 9962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
I20260928 02:04:29.133333 9962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 1 FOLLOWER]: Advancing to term 3
I20260928 02:04:29.137969 9962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 3.
I20260928 02:04:29.138526 9220 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters:
I20260928 02:04:29.138924 9988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 3 FOLLOWER]: Leader election won for term 3
W20260928 02:04:29.140269 9217 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:29.149140 10013 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:04:29.150046 9988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 3 LEADER]: Becoming Leader. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Running, Role: LEADER
I20260928 02:04:29.150660 9988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:29.155124 10010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 3 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:29.155556 10010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:29.156843 10015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:29.167586 10015 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:29.168349 10015 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:29.168861 10015 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:29.169870 9811 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:29.170081 9811 heartbeater.cc:503] Master 127.8.174.190:39911 was elected leader, sending a full tablet report...
I20260928 02:04:29.171178 10015 tablet_loader.cc:96] loaded metadata for tablet 006d1c27cc874f11bd66fb947942a96a (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:29.171881 10015 tablet_loader.cc:96] loaded metadata for tablet e02df1dfaba348449be5816040ea63bd (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:29.172247 10015 tablet_loader.cc:96] loaded metadata for tablet e65189e85b2641098a1bdfca68c049c4 (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:29.172549 10015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:29.173611 9952 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:54573; Asking this server to re-register.
I20260928 02:04:29.173863 10015 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:29.173936 10015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:29.174647 9811 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:29.175698 10015 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:29.176864 9951 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:29.176987 10015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Loaded TSK: 0
I20260928 02:04:29.178184 10015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:29.179515 10015 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:29.180696 10015 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:29.188820 9890 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:29.197518 9890 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:29.204394 9233 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46574:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:29.204408 9232 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46574:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:29.211195 9230 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:46574
I20260928 02:04:29.211143 9231 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46574:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:29.211413 9229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46574:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:29.213222 9228 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:46574
I20260928 02:04:29.213064 9225 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46574:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:29.213665 9227 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46574:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:29.214488 9224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46574:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:29.215340 9226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46574:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
F20260928 02:04:29.217774 9407 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7fb21bcfc9ad google::LogMessage::Fail() at ??:0
I20260928 02:04:29.220597 9962 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Refusing update from remote peer 5c9432f318fa44279928ebd2043a6640: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
W20260928 02:04:29.221082 9217 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 -> Peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Couldn't send request to peer 7017d177043645b5afe32c476055db7e. Status: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:04:29.221415 10010 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7fb21bcff88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb21bcfc4de google::LogMessage::Flush() at ??:0
@ 0x7fb21bcfc6d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb23189eba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260928 02:04:29.231400 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:29.231839 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:29.232234 9997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 3 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:29.232489 9997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:29.237145 9997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:29.237391 9997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fb23189d901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb2318a466c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fb2318a44e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260928 02:04:29.250237 9539 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:29.252398 9952 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:56223; Asking this server to re-register.
I20260928 02:04:29.253197 9539 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:29.255059 9952 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:29.254772 9988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:29.255354 9988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:29.255873 9230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Sending DeleteTablet for 3 replicas of tablet 006d1c27cc874f11bd66fb947942a96a
I20260928 02:04:29.256016 9228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Sending DeleteTablet for 3 replicas of tablet e02df1dfaba348449be5816040ea63bd
@ 0x7fb21e926eaf kudu::Thread::SuperviseThread() at ??:0
I20260928 02:04:29.258102 9230 catalog_manager.cc:6565] ReplaceTablet: tablet 006d1c27cc874f11bd66fb947942a96a deleted and replaced by tablet 4d4dc773cf234b2aa21013ec12b52290
@ 0x7fb221c006db start_thread at ??:0
@ 0x7fb21ab9661f clone at ??:0
I20260928 02:04:29.269469 10019 mvcc.cc:205] Tried to move back new op lower bound from 7334138139516751872 to 7334138134266966016. Current Snapshot: MvccSnapshot[applied={T|T < 7334138139516751872}]
W20260928 02:04:29.437274 10028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:29.259683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.8.174.130:39843)
0928 02:04:29.268850 (+ 9167us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.268856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.268877 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:04:29.268953 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.268958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.269039 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.269144 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.436105 (+166961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57246: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":9035,"thread_start_us":10515,"threads_started":1}
W20260928 02:04:29.437639 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:29.257743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.8.174.129:44359)
0928 02:04:29.267599 (+ 9856us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.267607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.267635 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:04:29.267734 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.267740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.267826 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.267950 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.436730 (+168780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46786: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":9693,"thread_start_us":9543,"threads_started":1}
W20260928 02:04:29.439936 10030 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:29.260424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.8.174.131:37953)
0928 02:04:29.268493 (+ 8069us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.268499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.439255 (+170756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55800: BlockingRecv error: recv got EOF from 127.0.0.1:55800 (error 108)
Metrics: {"server-negotiator.queue_time_us":7945,"thread_start_us":7851,"threads_started":1}
I20260928 02:04:29.625867 10040 consensus_queue.cc:1035] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.628531 10041 consensus_queue.cc:1035] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.630036 9676 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:29.631783 9952 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:35795; Asking this server to re-register.
I20260928 02:04:29.632555 9676 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:29.633951 9952 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:29.643438 10048 consensus_queue.cc:1035] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.646624 10053 consensus_queue.cc:1035] T 006d1c27cc874f11bd66fb947942a96a P 8c51770d4a61463c9330530e0d019d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.739228 10057 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:29.739459 10057 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:29.741323 10057 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:29.742473 9937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:29.743691 9936 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:29.743927 9937 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:29.744064 9937 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:29.744432 10057 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260928 02:04:29.772927 10048 consensus_queue.cc:1035] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.778679 10040 consensus_queue.cc:1035] T e02df1dfaba348449be5816040ea63bd P ebc8a50f334349b68dfbeb1ae9fc3635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:30.154150 10013 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:30.154261 10013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: loading cluster ID for follower catalog manager: success
I20260928 02:04:30.157115 10013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: acquiring CA information for follower catalog manager: success
I20260928 02:04:30.159281 10013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:30.363596 10071 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:30.363874 10071 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:30.364806 10071 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:30.367987 9936 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:30.368660 9937 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:30.368832 9937 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:30.369230 10071 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260928 02:04:31.372200 10084 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:31.372424 10084 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:31.373067 10084 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:31.375283 9937 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:31.375514 9936 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:31.375653 9936 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:31.375998 10084 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260928 02:04:32.063233 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--webserver_interface=127.8.174.189
--webserver_port=34795
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260928 02:04:32.294874 10098 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:32.295054 10098 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:32.295626 10098 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:32.300166 9937 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:32.300473 9936 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:32.300613 9936 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:32.300856 10098 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260928 02:04:32.417465 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:32.417917 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:32.418049 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:32.429530 10096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:32.429682 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:32.429752 10096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:32.429837 10096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:32.444051 10096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.174.189
--webserver_port=34795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:32.445761 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:32.448235 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:32.463214 10111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:32.463239 10109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:32.463419 10108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:32.464640 10096 server_base.cc:1034] running on GCE node
I20260928 02:04:32.465791 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:32.467857 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:32.469129 10096 hybrid_clock.cc:648] HybridClock initialized: now 1790561072469027 us; error 72 us; skew 500 ppm
I20260928 02:04:32.469560 10096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:32.478214 10096 webserver.cc:460] Webserver started at http://127.8.174.189:34795/ using document root <none> and password file <none>
I20260928 02:04:32.479089 10096 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:32.479229 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:32.485649 10096 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:04:32.489080 10117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:32.490914 10096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 02:04:32.491107 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:32.492146 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:32.526461 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:32.527573 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:32.527887 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:32.547256 10096 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.189:36913
I20260928 02:04:32.547277 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.189:36913 every 8 connection(s)
I20260928 02:04:32.548846 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
I20260928 02:04:32.554159 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 10096
I20260928 02:04:32.557188 10171 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:32.562593 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap starting.
I20260928 02:04:32.573475 10171 log.cc:824] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:32.591848 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=7 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:04:32.592432 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap complete.
I20260928 02:04:32.600729 10171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:32.601641 10171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Initialized, Role: FOLLOWER
I20260928 02:04:32.602223 10171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:32.603219 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:32.603609 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:32.604542 10171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:32.618217 10189 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:32.618369 10189 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: loading cluster ID for follower catalog manager: success
I20260928 02:04:32.622984 10189 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: acquiring CA information for follower catalog manager: success
I20260928 02:04:32.625820 10189 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:32.636966 10096 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:32.646498 10096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:32.807075 9677 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:32.810274 10136 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:33223; Asking this server to re-register.
I20260928 02:04:32.811121 9677 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:32.812783 10136 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:32.962415 9812 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:32.964128 10136 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:58443; Asking this server to re-register.
I20260928 02:04:32.964659 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:32.965934 10136 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:32.980165 10195 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:32.980386 10195 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:32.981148 10195 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:32.984360 9936 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:32.989253 10146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640" is_pre_election: true
I20260928 02:04:32.989892 10146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f09e70c2f7354c4e888b5aa6f0175a22 in term 3.
I20260928 02:04:32.990459 9937 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 7017d177043645b5afe32c476055db7e
I20260928 02:04:32.990778 10195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260928 02:04:32.990914 10195 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:32.991011 10195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 3 FOLLOWER]: Advancing to term 4
I20260928 02:04:32.993263 10195 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:32.993921 10195 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 election: Requested vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:32.994472 10146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640"
I20260928 02:04:32.994675 10146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 3 FOLLOWER]: Advancing to term 4
W20260928 02:04:32.995591 9936 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:32.999306 10146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f09e70c2f7354c4e888b5aa6f0175a22 in term 4.
I20260928 02:04:32.999830 9937 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 7017d177043645b5afe32c476055db7e
I20260928 02:04:33.000149 10195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 FOLLOWER]: Leader election won for term 4
I20260928 02:04:33.000602 10195 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 LEADER]: Becoming Leader. State: Replica: f09e70c2f7354c4e888b5aa6f0175a22, State: Running, Role: LEADER
I20260928 02:04:33.001044 10195 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:33.002754 10199 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09e70c2f7354c4e888b5aa6f0175a22. Latest consensus state: current_term: 4 leader_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:33.003069 10199 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:33.003662 10201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:33.006224 10201 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:33.006762 10201 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:33.007079 10201 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:33.009330 10201 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:33.009574 10201 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:33.009979 10201 tablet_loader.cc:96] loaded metadata for tablet e65189e85b2641098a1bdfca68c049c4 (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:33.010218 10201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:33.011183 10201 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:33.011257 10201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:33.012490 10201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:33.013370 10201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Loaded TSK: 0
I20260928 02:04:33.015317 10201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:33.016294 10201 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:33.017114 10201 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:33.021083 9540 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:33.022892 10136 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:33541; Asking this server to re-register.
I20260928 02:04:33.023514 9540 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:33.024883 10136 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:33.097167 9952 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:49276
I20260928 02:04:33.097417 9950 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:49276
I20260928 02:04:33.097167 9951 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:49276
I20260928 02:04:33.098479 9949 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.099035 9951 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.099114 9947 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.099363 9944 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49276:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
I20260928 02:04:33.099783 9943 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.099985 9945 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.100046 9946 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.098394 9948 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.101521 9950 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49276:
name: "default.Table_483ce76de8ea43dea7e133f72fa81aa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:33.102159 9952 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49276:
table { table_name: "default.Table_ccd4aeece7af4142a6e4c84a9bc39b89" } new_table_name: "default.Table_e5b9b953426e458a8839ebcc1ea8436f" modify_external_catalogs: true
F20260928 02:04:33.105698 10011 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7f48c5f199ad google::LogMessage::Fail() at ??:0
@ 0x7f48c5f1c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f48c5f194de google::LogMessage::Flush() at ??:0
@ 0x7f48c5f196d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7f48dbabbba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f48dbaba901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f48dbac166c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f48dbac14e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f48c8b43eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f48cbe1d6db start_thread at ??:0
@ 0x7f48c4db361f clone at ??:0
W20260928 02:04:33.300453 9566 connection.cc:535] client connection to 127.8.174.190:39911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:33.301386 9820 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.190:39911)
W20260928 02:04:33.301807 9820 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.190:39911)
W20260928 02:04:33.302503 9820 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.190:39911)
I20260928 02:04:33.733558 10219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:33.733788 10219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:33.735667 10219 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:33.736949 10121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:33.738566 10121 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:33.738788 10118 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:33.738977 10118 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640; no voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:33.739290 10219 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260928 02:04:34.322927 10228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:34.323122 10228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:34.323774 10228 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:34.326013 10118 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:34.326191 10121 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:34.326345 10121 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640; no voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:34.326592 10228 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260928 02:04:34.918637 10237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:34.918936 10237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:34.919655 10237 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:34.921561 10118 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:34.921713 10121 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:34.921878 10121 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640; no voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:34.922151 10237 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260928 02:04:35.645586 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.188:44075
--webserver_interface=127.8.174.188
--webserver_port=34749
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260928 02:04:35.981127 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:35.981447 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:35.981532 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:35.989907 10250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:35.990029 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:35.990082 10250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:35.990136 10250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:36.000041 10250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.188:44075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.174.188
--webserver_port=34749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:36.001438 10250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:36.003320 10250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:36.016330 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:36.016388 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:36.016312 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:36.017385 10250 server_base.cc:1034] running on GCE node
I20260928 02:04:36.018214 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:36.019730 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:36.020978 10250 hybrid_clock.cc:648] HybridClock initialized: now 1790561076020913 us; error 52 us; skew 500 ppm
I20260928 02:04:36.021298 10250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:36.024542 10250 webserver.cc:460] Webserver started at http://127.8.174.188:34749/ using document root <none> and password file <none>
I20260928 02:04:36.025279 10250 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:36.025398 10250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:36.031244 10250 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:04:36.034507 10268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:36.036250 10250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:36.036420 10250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
uuid: "7017d177043645b5afe32c476055db7e"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:36.037528 10250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:36.052435 10250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:36.053392 10250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:36.053676 10250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:36.073683 10250 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.188:44075
I20260928 02:04:36.073731 10319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.188:44075 every 8 connection(s)
I20260928 02:04:36.075341 10250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-2/data/info.pb
I20260928 02:04:36.082458 10320 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:36.083302 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 10250
I20260928 02:04:36.086678 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Bootstrap starting.
I20260928 02:04:36.097750 10320 log.cc:824] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:36.113484 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=10 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:04:36.114005 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Bootstrap complete.
I20260928 02:04:36.121884 10320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:36.122915 10320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7017d177043645b5afe32c476055db7e, State: Initialized, Role: FOLLOWER
I20260928 02:04:36.123574 10320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:36.124398 10324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:36.124733 10324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:36.125554 10320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:36.135751 10250 master_runner.cc:186] Error getting master registration for 127.8.174.190:39911: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111),
I20260928 02:04:36.143424 10250 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:36.153172 10338 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:36.153316 10338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: loading cluster ID for follower catalog manager: success
I20260928 02:04:36.156282 10338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: acquiring CA information for follower catalog manager: success
I20260928 02:04:36.159145 10338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:36.312158 9675 heartbeater.cc:347] Connected to a master server at 127.8.174.188:44075
I20260928 02:04:36.315948 10285 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:57149; Asking this server to re-register.
I20260928 02:04:36.316931 9675 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:36.319093 10285 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:36.473261 9810 heartbeater.cc:347] Connected to a master server at 127.8.174.188:44075
I20260928 02:04:36.475317 10285 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:49569; Asking this server to re-register.
I20260928 02:04:36.475942 9810 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:36.477572 10285 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:36.532752 9538 heartbeater.cc:347] Connected to a master server at 127.8.174.188:44075
I20260928 02:04:36.534328 10285 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:35213; Asking this server to re-register.
I20260928 02:04:36.534981 9538 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:36.536340 10285 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:36.584492 10348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:36.584682 10348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:36.585335 10348 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:36.588042 10121 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:36.592815 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e" is_pre_election: true
I20260928 02:04:36.593356 10295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 2.
I20260928 02:04:36.593907 10118 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e; no voters: f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:36.594177 10348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260928 02:04:36.594332 10348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:36.594424 10348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 4 FOLLOWER]: Advancing to term 5
I20260928 02:04:36.596426 10348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:36.597033 10348 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:36.597651 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e"
I20260928 02:04:36.597982 10295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 2 FOLLOWER]: Advancing to term 5
W20260928 02:04:36.598417 10121 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:36.602474 10295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 5.
I20260928 02:04:36.602950 10118 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e; no voters: f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:36.603356 10348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Leader election won for term 5
I20260928 02:04:36.603981 10348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 LEADER]: Becoming Leader. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Running, Role: LEADER
I20260928 02:04:36.604513 10348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:36.607331 10352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 5 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:36.607733 10352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:36.608372 10354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:36.610790 10354 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:36.611325 10354 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:36.611696 10354 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:36.613624 10354 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:36.613866 10354 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:36.614252 10354 tablet_loader.cc:96] loaded metadata for tablet e65189e85b2641098a1bdfca68c049c4 (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:36.614524 10354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:36.615412 10354 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:36.615482 10354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:36.616658 10354 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:36.617486 10354 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Loaded TSK: 0
I20260928 02:04:36.618418 10354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:36.619252 10354 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:36.620023 10354 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:36.626602 10135 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:54072
I20260928 02:04:36.626758 10136 master_service.cc:946] ReplaceTablet: received request to replace tablet e65189e85b2641098a1bdfca68c049c4 from {username='slave'} at 127.0.0.1:54072
I20260928 02:04:36.627633 10131 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:36.628085 10128 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:36.628208 10132 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:36.628749 10133 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:36.628906 10130 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:36.628898 10135 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_483ce76de8ea43dea7e133f72fa81aa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:36.629298 10127 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:36.630116 10129 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54072:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
I20260928 02:04:36.627578 10134 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260928 02:04:36.634325 10188 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
I20260928 02:04:36.635955 10295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Refusing update from remote peer 5c9432f318fa44279928ebd2043a6640: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
@ 0x7f9ead65a9ad google::LogMessage::Fail() at ??:0
W20260928 02:04:36.636641 10121 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 -> Peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Couldn't send request to peer f09e70c2f7354c4e888b5aa6f0175a22. Status: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:04:36.636914 10348 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.000s
@ 0x7f9ead65d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9ead65a4de google::LogMessage::Flush() at ??:0
@ 0x7f9ead65a6d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9ec31fcba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260928 02:04:36.650952 10357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 5 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:36.651264 10357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9ec31fb901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260928 02:04:36.651779 10355 mvcc.cc:205] Tried to move back new op lower bound from 7334138169889136640 to 7334138169775960064. Current Snapshot: MvccSnapshot[applied={T|T < 7334138169889136640}]
I20260928 02:04:36.651597 10361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:36.651912 10361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:36.653079 10361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:36.653294 10361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9ec320266c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260928 02:04:36.655792 10348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:36.656064 10348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:36.657354 10136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Sending DeleteTablet for 3 replicas of tablet e65189e85b2641098a1bdfca68c049c4
@ 0x7f9ec32024e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9eb0284eaf kudu::Thread::SuperviseThread() at ??:0
I20260928 02:04:36.666966 10136 catalog_manager.cc:6565] ReplaceTablet: tablet e65189e85b2641098a1bdfca68c049c4 deleted and replaced by tablet dbff92b3afb64048bb1dda2517c82f7a
I20260928 02:04:36.667521 10136 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54072:
table { table_name: "default.Table_ccd4aeece7af4142a6e4c84a9bc39b89" } new_table_name: "default.Table_e5b9b953426e458a8839ebcc1ea8436f" modify_external_catalogs: true
@ 0x7f9eb355e6db start_thread at ??:0
I20260928 02:04:36.675573 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet e65189e85b2641098a1bdfca68c049c4 with delete_type TABLET_DATA_DELETED (replaced by tablet dbff92b3afb64048bb1dda2517c82f7a) from {username='slave'} at 127.0.0.1:34538
@ 0x7f9eac4f461f clone at ??:0
I20260928 02:04:36.675925 9471 tablet_service.cc:1467] Processing CreateTablet for tablet 54a8ea3c4dee42d09d5938e9202df82f (DEFAULT_TABLE table=default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:36.676245 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4dc773cf234b2aa21013ec12b52290 (DEFAULT_TABLE table=default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:36.676456 9471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a8ea3c4dee42d09d5938e9202df82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:36.676714 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4dc773cf234b2aa21013ec12b52290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:36.687410 10373 tablet_replica.cc:330] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: stopping tablet replica
I20260928 02:04:36.687649 10373 raft_consensus.cc:2229] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:36.688395 10373 raft_consensus.cc:2258] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:36.689164 10373 ts_tablet_manager.cc:1905] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:36.690743 10374 tablet_bootstrap.cc:492] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635: Bootstrap starting.
I20260928 02:04:36.692250 10374 tablet_bootstrap.cc:654] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:36.694833 10373 ts_tablet_manager.cc:1918] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260928 02:04:36.694979 10373 log.cc:1196] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: Deleting WAL directory at /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-0/wal/wals/e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:36.695415 10373 ts_tablet_manager.cc:1939] T e65189e85b2641098a1bdfca68c049c4 P ebc8a50f334349b68dfbeb1ae9fc3635: Deleting consensus metadata
I20260928 02:04:36.695809 10374 tablet_bootstrap.cc:492] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635: No bootstrap required, opened a new log
I20260928 02:04:36.695961 10374 ts_tablet_manager.cc:1397] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 02:04:36.696838 10374 raft_consensus.cc:348] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:36.697093 10374 raft_consensus.cc:374] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:36.697185 10374 raft_consensus.cc:729] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Initialized, Role: FOLLOWER
I20260928 02:04:36.697428 10374 consensus_queue.cc:260] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:36.698006 10374 ts_tablet_manager.cc:1428] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:04:36.698372 10374 tablet_bootstrap.cc:492] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635: Bootstrap starting.
I20260928 02:04:36.700081 10374 tablet_bootstrap.cc:654] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:36.702229 10374 tablet_bootstrap.cc:492] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635: No bootstrap required, opened a new log
I20260928 02:04:36.702406 10374 ts_tablet_manager.cc:1397] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260928 02:04:36.703135 10374 raft_consensus.cc:348] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:04:36.703377 10374 raft_consensus.cc:374] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:36.703444 10374 raft_consensus.cc:729] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc8a50f334349b68dfbeb1ae9fc3635, State: Initialized, Role: FOLLOWER
I20260928 02:04:36.703672 10374 consensus_queue.cc:260] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:04:36.704226 10374 ts_tablet_manager.cc:1428] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20260928 02:04:36.863332 9426 connection.cc:535] server connection from 127.0.0.1:34538 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:36.863683 9427 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:36.863855 10369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:36.667495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.8.174.131:37953)
0928 02:04:36.686864 (+ 19369us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:36.686871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:36.686898 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:04:36.686977 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:36.686983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:36.687071 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:36.687184 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:36.863480 (+176296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45818: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":19198,"thread_start_us":22385,"threads_started":1}
W20260928 02:04:36.863853 10372 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:36.668420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.8.174.130:39843)
0928 02:04:36.669048 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:36.669055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:36.673580 (+ 4525us) server_negotiation.cc:407] Connection header received
0928 02:04:36.673753 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:36.673758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:36.673842 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:36.673951 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:36.674633 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:36.683054 (+ 8421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:36.863485 (+180431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48462: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":578,"threads_started":1}
W20260928 02:04:36.864418 9818 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:36.865175 9818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.189:36913)
W20260928 02:04:36.865691 9818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:36.865907 9564 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:36.865970 10347 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:36.864543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.174.189:36913 (local address 127.8.174.129:50003)
0928 02:04:36.864863 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:36.864884 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:36.865005 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:36.865753 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":184}
W20260928 02:04:36.866377 9818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.189:36913)
I20260928 02:04:37.166103 10383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:37.166353 10383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:37.168202 10383 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:37.169193 10272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:37.170372 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:37.170670 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:37.170902 10272 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:37.171339 10383 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260928 02:04:37.297981 10216 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:37.298216 10216 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:37.299077 10216 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:37.300247 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:04:37.300496 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:04:37.300621 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:37.300889 10216 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:04:37.449188 10216 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:37.449388 10216 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:04:37.449993 10216 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:37.450999 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:04:37.451179 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:04:37.451308 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:37.451555 10216 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:04:37.786621 10388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:37.786861 10388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:37.787637 10388 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:37.790616 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:37.790861 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:37.791030 10270 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:37.791322 10388 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260928 02:04:38.490559 10393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:38.490759 10393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:38.491411 10393 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:38.493530 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:38.493762 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:38.493877 10272 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:38.494180 10393 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260928 02:04:39.032596 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--webserver_interface=127.8.174.189
--webserver_port=34795
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260928 02:04:39.275059 10400 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ebc8a50f334349b68dfbeb1ae9fc3635)
I20260928 02:04:39.275281 10400 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:39.276993 10400 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:39.287659 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65189e85b2641098a1bdfca68c049c4" candidate_uuid: "53ad00df981d4266a118559f648b1abe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c51770d4a61463c9330530e0d019d98" is_pre_election: true
W20260928 02:04:39.288631 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:39.288837 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:04:39.289247 10400 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 02:04:39.376338 10399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:39.376646 10399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:39.376757 10399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:39.384240 10399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:39.384371 10399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:39.384419 10399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:39.384491 10399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:39.394517 10399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.174.189
--webserver_port=34795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:39.395927 10399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:39.397694 10399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:39.409608 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:39.409694 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:39.409642 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:39.411055 10399 server_base.cc:1034] running on GCE node
I20260928 02:04:39.411939 10399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:39.413460 10399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:39.414695 10399 hybrid_clock.cc:648] HybridClock initialized: now 1790561079414624 us; error 45 us; skew 500 ppm
I20260928 02:04:39.415067 10399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:39.423640 10399 webserver.cc:460] Webserver started at http://127.8.174.189:34795/ using document root <none> and password file <none>
I20260928 02:04:39.424512 10399 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:39.424644 10399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:39.430702 10399 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:04:39.433924 10417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:39.435444 10399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:39.435631 10399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:39.436638 10399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:39.457908 10399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:39.458920 10399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:39.459205 10399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:39.477533 10399 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.189:36913
I20260928 02:04:39.477563 10468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.189:36913 every 8 connection(s)
I20260928 02:04:39.479138 10399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
I20260928 02:04:39.480671 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 10399
I20260928 02:04:39.488131 10469 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:39.493902 10469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap starting.
I20260928 02:04:39.500875 9812 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:39.507227 10469 log.cc:824] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:39.526271 10469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=19 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260928 02:04:39.526958 10469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap complete.
I20260928 02:04:39.534801 10469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:39.538481 10469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Initialized, Role: FOLLOWER
I20260928 02:04:39.539239 10469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:39.540051 10476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:39.540465 10476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:39.541316 10469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:39.554831 10490 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:39.555058 10490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: loading cluster ID for follower catalog manager: success
I20260928 02:04:39.559501 10490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: acquiring CA information for follower catalog manager: success
I20260928 02:04:39.573123 10490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:39.571764 10399 master_runner.cc:186] Error getting master registration for 127.8.174.190:39911: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111),
I20260928 02:04:39.581059 10399 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:39.612466 10495 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ebc8a50f334349b68dfbeb1ae9fc3635)
I20260928 02:04:39.612654 10495 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:39.613320 10495 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:39.613859 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65189e85b2641098a1bdfca68c049c4" candidate_uuid: "8c51770d4a61463c9330530e0d019d98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe" is_pre_election: true
I20260928 02:04:39.614125 9630 raft_consensus.cc:2454] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c51770d4a61463c9330530e0d019d98 in term 1.
W20260928 02:04:39.614187 9700 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:39.614682 9698 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98; no voters: ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:04:39.614928 10495 raft_consensus.cc:2781] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:04:39.615091 10495 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ebc8a50f334349b68dfbeb1ae9fc3635)
I20260928 02:04:39.615206 10495 raft_consensus.cc:3037] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:04:39.617295 10495 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:39.617839 10495 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 2 election: Requested vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:39.618399 9630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65189e85b2641098a1bdfca68c049c4" candidate_uuid: "8c51770d4a61463c9330530e0d019d98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53ad00df981d4266a118559f648b1abe"
I20260928 02:04:39.618649 9630 raft_consensus.cc:3037] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 1 FOLLOWER]: Advancing to term 2
W20260928 02:04:39.618681 9700 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:39.621477 9630 raft_consensus.cc:2454] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c51770d4a61463c9330530e0d019d98 in term 2.
I20260928 02:04:39.622004 9698 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98; no voters: ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:04:39.622363 10495 raft_consensus.cc:2781] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:04:39.622526 10495 raft_consensus.cc:686] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 2 LEADER]: Becoming Leader. State: Replica: 8c51770d4a61463c9330530e0d019d98, State: Running, Role: LEADER
I20260928 02:04:39.622920 10495 consensus_queue.cc:237] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:39.628033 10434 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:55863; Asking this server to re-register.
I20260928 02:04:39.628787 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:39.630689 10434 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:39.775719 10498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:39.775915 10498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:39.776647 10498 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:39.779422 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:39.785998 10444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7017d177043645b5afe32c476055db7e" candidate_term: 6 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640" is_pre_election: true
I20260928 02:04:39.786795 10444 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7017d177043645b5afe32c476055db7e for term 6 because replica has last-logged OpId of term: 5 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 19.
I20260928 02:04:39.787812 10270 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:39.788360 10498 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260928 02:04:39.881799 9677 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:39.882728 9540 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:39.883564 10433 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:37935; Asking this server to re-register.
I20260928 02:04:39.884214 10434 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:33399; Asking this server to re-register.
I20260928 02:04:39.884248 9677 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:39.884796 9540 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:39.885848 10434 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:39.886209 10433 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:39.929193 10476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:39.929520 10476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:39.931555 10476 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:39.932330 10421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:39.934669 10421 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:39.939467 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e" is_pre_election: true
I20260928 02:04:39.939783 10295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 5.
I20260928 02:04:39.940521 10418 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e; no voters: f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:39.940985 10476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260928 02:04:39.941170 10476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:39.941277 10476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 5 FOLLOWER]: Advancing to term 6
I20260928 02:04:39.945542 10476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:39.946336 10476 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 6 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:39.947073 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e"
I20260928 02:04:39.947376 10295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 5 FOLLOWER]: Advancing to term 6
W20260928 02:04:39.948050 10421 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:39.949548 10295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 6.
I20260928 02:04:39.950085 10418 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e; no voters: f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:39.950476 10476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 FOLLOWER]: Leader election won for term 6
I20260928 02:04:39.951006 10476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 LEADER]: Becoming Leader. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Running, Role: LEADER
I20260928 02:04:39.951522 10476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:39.954645 10505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 6 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:39.955096 10505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:40.341879 10295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Refusing update from remote peer 5c9432f318fa44279928ebd2043a6640: Log matching property violated. Preceding OpId in replica: term: 5 index: 19. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
I20260928 02:04:40.342674 10505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
W20260928 02:04:40.346589 10421 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 -> Peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Couldn't send request to peer f09e70c2f7354c4e888b5aa6f0175a22. Status: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:04:40.346526 10509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 6 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:40.346828 10509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:40.348658 10505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:40.348939 10505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:40.350857 10509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:40.351118 10509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:40.369567 10507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:40.372145 10507 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:40.372617 10507 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:40.372902 10507 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:40.374795 10507 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:40.375051 10507 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:40.375278 10507 tablet_loader.cc:96] loaded metadata for tablet dbff92b3afb64048bb1dda2517c82f7a (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:40.375727 10507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:40.376556 10507 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:40.376626 10507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:40.377760 10507 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:40.378594 10507 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Loaded TSK: 0
I20260928 02:04:40.379453 10507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:40.380230 10507 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:40.381023 10507 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:40.457652 10434 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:54724
I20260928 02:04:40.457911 10433 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:54724
I20260928 02:04:40.457976 10432 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:54724
I20260928 02:04:40.459146 10427 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:40.459487 10426 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54724:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
I20260928 02:04:40.459715 10434 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:40.460040 10425 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:40.460117 10429 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:40.460671 10431 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:40.460793 10428 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:40.461445 10433 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_483ce76de8ea43dea7e133f72fa81aa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:40.461577 10432 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54724:
table { table_name: "default.Table_ccd4aeece7af4142a6e4c84a9bc39b89" } new_table_name: "default.Table_e5b9b953426e458a8839ebcc1ea8436f" modify_external_catalogs: true
I20260928 02:04:40.461367 10430 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54724:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260928 02:04:40.465380 10489 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7ffa192849ad google::LogMessage::Fail() at ??:0
@ 0x7ffa1928788a google::LogMessage::SendToLog() at ??:0
@ 0x7ffa192844de google::LogMessage::Flush() at ??:0
@ 0x7ffa192846d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7ffa2ee26ba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7ffa2ee25901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ffa2ee2c66c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7ffa2ee2c4e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ffa1beaeeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffa1f1886db start_thread at ??:0
@ 0x7ffa1811e61f clone at ??:0
I20260928 02:04:40.621992 10494 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:40.622190 10494 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:04:40.622953 10494 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:40.623903 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:04:40.624079 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:04:40.624207 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:40.624444 10494 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260928 02:04:40.667311 9699 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:40.667778 9818 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:40.668407 9818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.189:36913)
W20260928 02:04:40.669291 9818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:40.670077 9818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:40.679384 9700 consensus_peers.cc:487] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 -> Peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Couldn't send request to peer ebc8a50f334349b68dfbeb1ae9fc3635. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:04:40.682803 9630 raft_consensus.cc:1261] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Refusing update from remote peer 8c51770d4a61463c9330530e0d019d98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260928 02:04:40.683456 10526 consensus_queue.cc:1035] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260928 02:04:40.685043 9677 heartbeater.cc:503] Master 127.8.174.189:36913 was elected leader, sending a full tablet report...
I20260928 02:04:40.689016 10494 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:40.689193 10494 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:40.689846 10494 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:40.691045 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:04:40.691269 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:04:40.691404 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:40.691733 10494 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:04:40.849315 10509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:40.849521 10509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:40.850229 10509 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:40.852783 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:40.852998 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:40.853127 10272 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:40.853387 10509 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260928 02:04:40.889513 9540 heartbeater.cc:503] Master 127.8.174.189:36913 was elected leader, sending a full tablet report...
I20260928 02:04:41.144773 10250 master_runner.cc:186] Error getting master registration for 127.8.174.190:39911: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111),
I20260928 02:04:41.144986 10250 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:41.370601 10541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:41.370791 10541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:41.371459 10541 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:41.373160 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:41.373322 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
I20260928 02:04:41.373471 10272 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:41.373693 10541 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260928 02:04:41.901003 10548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:41.901186 10548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:41.901810 10548 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:41.903714 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:41.903935 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:41.904068 10270 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:41.904424 10548 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260928 02:04:42.366936 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.190:39911
--webserver_interface=127.8.174.190
--webserver_port=39585
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260928 02:04:42.632160 10559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:42.632381 10559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:42.633039 10559 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:42.633973 10272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20260928 02:04:42.635334 10272 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911): Network error: Client connection negotiation failed: client connection to 127.8.174.190:39911: connect: Connection refused (error 111)
W20260928 02:04:42.635530 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:42.635653 10270 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7017d177043645b5afe32c476055db7e; no voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:42.635905 10559 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260928 02:04:42.722781 10556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:42.723177 10556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:42.723282 10556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:42.731688 10556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:42.731820 10556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:42.731891 10556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:42.731966 10556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:42.742502 10556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.190:39911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.174.190
--webserver_port=39585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:42.743893 10556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:42.745682 10556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:42.757346 10573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:42.757323 10576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:42.757318 10574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:42.758075 10556 server_base.cc:1034] running on GCE node
I20260928 02:04:42.758970 10556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:42.760406 10556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:42.761615 10556 hybrid_clock.cc:648] HybridClock initialized: now 1790561082761543 us; error 58 us; skew 500 ppm
I20260928 02:04:42.761941 10556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:42.764317 10556 webserver.cc:460] Webserver started at http://127.8.174.190:39585/ using document root <none> and password file <none>
I20260928 02:04:42.765002 10556 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:42.765096 10556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:42.770597 10556 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:04:42.773641 10582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:42.775326 10556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:42.775519 10556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-3l00"
I20260928 02:04:42.776575 10556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:42.802196 10556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:42.803226 10556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:42.803541 10556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:42.823518 10556 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.190:39911
I20260928 02:04:42.823536 10633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.190:39911 every 8 connection(s)
I20260928 02:04:42.825218 10556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:42.825588 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 10556
I20260928 02:04:42.833655 10634 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:42.838500 10634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Bootstrap starting.
I20260928 02:04:42.848387 10634 log.cc:824] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:42.864931 10634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=7 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260928 02:04:42.865525 10634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: Bootstrap complete.
I20260928 02:04:42.874954 10634 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:42.876592 10634 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09e70c2f7354c4e888b5aa6f0175a22, State: Initialized, Role: FOLLOWER
I20260928 02:04:42.877377 10634 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:42.878247 10638 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:42.878734 10638 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:42.879715 10634 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:42.891594 10652 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:42.891837 10652 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: loading cluster ID for follower catalog manager: success
I20260928 02:04:42.900353 10652 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: acquiring CA information for follower catalog manager: success
I20260928 02:04:42.905819 10652 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:42.907614 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:04:42.913596 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:43.207798 9676 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:43.211134 10599 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:35543; Asking this server to re-register.
I20260928 02:04:43.211918 9676 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:43.213748 10599 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:43.226598 9811 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:43.228566 10599 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:53249; Asking this server to re-register.
I20260928 02:04:43.229302 9811 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:43.230813 10599 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:43.232532 9539 heartbeater.cc:347] Connected to a master server at 127.8.174.190:39911
I20260928 02:04:43.234033 10599 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:44459; Asking this server to re-register.
I20260928 02:04:43.234778 9539 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:43.236065 10599 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:43.287215 10638 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:43.287520 10638 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:43.289577 10638 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:43.290387 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:43.291970 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:43.295552 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e" is_pre_election: true
I20260928 02:04:43.295894 10295 raft_consensus.cc:2354] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f09e70c2f7354c4e888b5aa6f0175a22 for earlier term 5. Current term is 6.
I20260928 02:04:43.296631 10583 leader_election.cc:398] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f09e70c2f7354c4e888b5aa6f0175a22 for earlier term 5. Current term is 6.
I20260928 02:04:43.296710 10583 leader_election.cc:401] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260928 02:04:43.297071 10638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 4 FOLLOWER]: Advancing to term 6
I20260928 02:04:43.301855 10638 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f09e70c2f7354c4e888b5aa6f0175a22 for earlier term 5. Current term is 6.
I20260928 02:04:44.043541 10669 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:44.043838 10669 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:44.044593 10669 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:44.045195 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 7 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e" is_pre_election: true
I20260928 02:04:44.045538 10295 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f09e70c2f7354c4e888b5aa6f0175a22 for term 7 because replica has last-logged OpId of term: 6 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 17.
W20260928 02:04:44.046804 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:44.046921 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:44.047258 10669 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260928 02:04:44.683924 10678 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:44.684206 10678 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:44.685177 10678 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:44.685475 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 7 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "7017d177043645b5afe32c476055db7e" is_pre_election: true
I20260928 02:04:44.685850 10295 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f09e70c2f7354c4e888b5aa6f0175a22 for term 7 because replica has last-logged OpId of term: 6 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 17.
W20260928 02:04:44.687239 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:44.687343 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:44.687577 10678 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260928 02:04:44.979204 10684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:44.979409 10684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:44.980126 10684 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
W20260928 02:04:44.982813 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:44.987656 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7017d177043645b5afe32c476055db7e" candidate_term: 7 candidate_status { last_received { term: 6 index: 21 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:44.988152 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7017d177043645b5afe32c476055db7e in term 6.
I20260928 02:04:44.988869 10272 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640
I20260928 02:04:44.989214 10684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260928 02:04:44.989352 10684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5c9432f318fa44279928ebd2043a6640)
I20260928 02:04:44.989452 10684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 6 FOLLOWER]: Advancing to term 7
I20260928 02:04:44.991694 10684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:44.992437 10684 leader_election.cc:288] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913)
I20260928 02:04:44.993057 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7017d177043645b5afe32c476055db7e" candidate_term: 7 candidate_status { last_received { term: 6 index: 21 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
I20260928 02:04:44.993353 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 6 FOLLOWER]: Advancing to term 7
W20260928 02:04:44.993985 10270 leader_election.cc:334] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:44.995886 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7017d177043645b5afe32c476055db7e in term 7.
I20260928 02:04:44.996358 10272 leader_election.cc:302] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640
I20260928 02:04:44.996658 10684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 7 FOLLOWER]: Leader election won for term 7
I20260928 02:04:44.997174 10684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [term 7 LEADER]: Becoming Leader. State: Replica: 7017d177043645b5afe32c476055db7e, State: Running, Role: LEADER
I20260928 02:04:44.997625 10684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 6.21, Last appended by leader: 21, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:44.999452 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7017d177043645b5afe32c476055db7e. Latest consensus state: current_term: 7 leader_uuid: "7017d177043645b5afe32c476055db7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:44.999933 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:45.000687 10691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:45.003427 10691 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:45.003924 10691 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:45.004215 10691 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:45.006168 10691 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:45.006532 10691 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:45.006798 10691 tablet_loader.cc:96] loaded metadata for tablet dbff92b3afb64048bb1dda2517c82f7a (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:45.007292 10691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:45.008215 10691 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:45.008294 10691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:45.009459 10691 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:45.010270 10691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7017d177043645b5afe32c476055db7e: Loaded TSK: 0
I20260928 02:04:45.011183 10691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:45.011999 10691 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:45.012740 10691 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:45.040321 9744 tablet_service.cc:1514] Processing DeleteTablet for tablet e65189e85b2641098a1bdfca68c049c4 with delete_type TABLET_DATA_DELETED (replaced by tablet dbff92b3afb64048bb1dda2517c82f7a) from {username='slave'} at 127.0.0.1:44708
I20260928 02:04:45.041030 10695 tablet_replica.cc:330] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: stopping tablet replica
I20260928 02:04:45.041270 10695 raft_consensus.cc:2229] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 2 LEADER]: Raft consensus shutting down.
I20260928 02:04:45.041812 10695 raft_consensus.cc:2258] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:45.042464 10695 ts_tablet_manager.cc:1905] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:45.047132 10695 ts_tablet_manager.cc:1918] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260928 02:04:45.047276 10695 log.cc:1196] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: Deleting WAL directory at /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/ts-2/wal/wals/e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:45.047610 10695 ts_tablet_manager.cc:1939] T e65189e85b2641098a1bdfca68c049c4 P 8c51770d4a61463c9330530e0d019d98: Deleting consensus metadata
I20260928 02:04:45.048714 10272 catalog_manager.cc:4841] TS 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): tablet e65189e85b2641098a1bdfca68c049c4 (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]) successfully deleted
I20260928 02:04:45.057744 10284 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:43962
I20260928 02:04:45.057876 10283 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:43962
I20260928 02:04:45.058271 10282 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:43962
I20260928 02:04:45.058492 10281 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:43962
I20260928 02:04:45.059120 10285 master_service.cc:946] ReplaceTablet: received request to replace tablet e65189e85b2641098a1bdfca68c049c4 from {username='slave'} at 127.0.0.1:43962
I20260928 02:04:45.059584 10283 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:45.059862 10278 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:45.060369 10277 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43962:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
I20260928 02:04:45.059520 10280 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:45.060648 10276 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_483ce76de8ea43dea7e133f72fa81aa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:45.062098 10285 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43962:
table { table_name: "default.Table_ccd4aeece7af4142a6e4c84a9bc39b89" } new_table_name: "default.Table_e5b9b953426e458a8839ebcc1ea8436f" modify_external_catalogs: true
I20260928 02:04:45.061452 10279 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:45.060796 10282 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:45.061530 10281 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:45.062122 10284 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43962:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260928 02:04:45.066076 10337 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7f7731fff9ad google::LogMessage::Fail() at ??:0
@ 0x7f773200288a google::LogMessage::SendToLog() at ??:0
@ 0x7f7731fff4de google::LogMessage::Flush() at ??:0
@ 0x7f7731fff6d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7f7747ba1ba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f7747ba0901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f7747ba766c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7747ba74e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7734c29eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7737f036db start_thread at ??:0
@ 0x7f7730e9961f clone at ??:0
W20260928 02:04:45.263052 9563 connection.cc:535] client connection to 127.8.174.188:44075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:45.263315 9817 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.188:44075)
W20260928 02:04:45.263602 9817 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.188:44075)
W20260928 02:04:45.264634 9426 connection.cc:535] client connection to 127.8.174.188:44075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:45.267140 9817 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.188:44075)
I20260928 02:04:45.535593 9810 heartbeater.cc:503] Master 127.8.174.188:44075 was elected leader, sending a full tablet report...
I20260928 02:04:45.605381 10713 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:45.605577 10713 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:45.606271 10713 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:45.608554 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:04:45.608733 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:45.608879 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:45.609125 10713 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260928 02:04:46.235387 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--webserver_interface=127.8.174.189
--webserver_port=34795
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260928 02:04:46.365609 10719 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:46.365972 10719 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:04:46.367349 10719 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:46.369091 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:04:46.370549 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:04:46.370672 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:46.370939 10719 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:04:46.455535 10719 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:46.455742 10719 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:46.456398 10719 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:46.457363 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:04:46.457564 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:04:46.457680 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:46.457991 10719 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260928 02:04:46.582965 10718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:46.583309 10718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:46.583424 10718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:46.590960 10718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:46.591086 10718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:46.591156 10718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:46.591231 10718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:46.601428 10718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.174.189
--webserver_port=34795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:46.602990 10718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:46.604925 10718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:46.617271 10724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:46.617208 10725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:46.617277 10727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:46.618065 10718 server_base.cc:1034] running on GCE node
I20260928 02:04:46.618960 10718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:46.620507 10718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:46.621750 10718 hybrid_clock.cc:648] HybridClock initialized: now 1790561086621673 us; error 47 us; skew 500 ppm
I20260928 02:04:46.622107 10718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:46.624562 10718 webserver.cc:460] Webserver started at http://127.8.174.189:34795/ using document root <none> and password file <none>
I20260928 02:04:46.625298 10718 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:46.625415 10718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:46.631376 10718 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:04:46.634766 10733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:46.636333 10718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:46.636535 10718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:46.637573 10718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:46.659271 10718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:46.660198 10718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:46.660485 10718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:46.678092 10718 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.189:36913
I20260928 02:04:46.678097 10784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.189:36913 every 8 connection(s)
I20260928 02:04:46.679667 10718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
I20260928 02:04:46.684868 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 10718
I20260928 02:04:46.687144 10785 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:46.692070 10785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap starting.
I20260928 02:04:46.702644 10785 log.cc:824] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:46.717696 9677 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:46.717909 9677 heartbeater.cc:503] Master 127.8.174.189:36913 was elected leader, sending a full tablet report...
I20260928 02:04:46.723726 10785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap replayed 1/1 log segments. Stats: ops{read=21 overwritten=0 applied=21 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:04:46.724242 10785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap complete.
I20260928 02:04:46.731694 10785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:46.732649 10785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Initialized, Role: FOLLOWER
I20260928 02:04:46.733328 10785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 6.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:46.734139 10789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:46.734522 10789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:46.735316 10785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:46.756773 10803 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:46.760600 10803 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: loading cluster ID for follower catalog manager: success
I20260928 02:04:46.772173 10803 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: acquiring CA information for follower catalog manager: success
I20260928 02:04:46.777441 10803 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:46.793462 10718 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:46.799661 10718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:46.912678 9540 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:46.913009 9540 heartbeater.cc:503] Master 127.8.174.189:36913 was elected leader, sending a full tablet report...
I20260928 02:04:46.916360 10750 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:57871; Asking this server to re-register.
I20260928 02:04:46.917114 9540 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:46.919126 10750 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:47.043695 9812 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:47.045053 10750 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:56669; Asking this server to re-register.
I20260928 02:04:47.045593 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:47.046957 10750 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:47.220474 9677 heartbeater.cc:503] Master 127.8.174.189:36913 was elected leader, sending a full tablet report...
I20260928 02:04:47.222203 10750 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:55389; Asking this server to re-register.
I20260928 02:04:47.222904 9677 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:47.224282 10750 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:47.228507 10789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:47.228786 10789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:47.231096 10789 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:47.231469 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 7 candidate_status { last_received { term: 6 index: 21 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:47.231798 10609 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9432f318fa44279928ebd2043a6640 in current term 7: Already voted for candidate 7017d177043645b5afe32c476055db7e in this term.
W20260928 02:04:47.231853 10734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:47.233269 10734 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:47.233457 10734 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640; no voters: 7017d177043645b5afe32c476055db7e, f09e70c2f7354c4e888b5aa6f0175a22
I20260928 02:04:47.233909 10789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 6 FOLLOWER]: Advancing to term 7
I20260928 02:04:47.238411 10789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260928 02:04:47.436127 10812 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:04:47.436379 10812 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:47.437112 10812 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:47.438014 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:04:47.438186 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:47.438354 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:04:47.438648 10812 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:04:47.896631 10817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:47.896836 10817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:47.897603 10817 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:47.898144 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 8 candidate_status { last_received { term: 6 index: 21 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:47.898429 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 7.
I20260928 02:04:47.898919 10737 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters:
I20260928 02:04:47.899243 10817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260928 02:04:47.899456 10817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:47.899576 10817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 7 FOLLOWER]: Advancing to term 8
W20260928 02:04:47.899997 10734 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:47.901793 10817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:47.902400 10817 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 8 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:47.902897 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 8 candidate_status { last_received { term: 6 index: 21 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
I20260928 02:04:47.903142 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 7 FOLLOWER]: Advancing to term 8
W20260928 02:04:47.903669 10734 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:47.905063 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 8.
I20260928 02:04:47.905426 10737 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 7017d177043645b5afe32c476055db7e
I20260928 02:04:47.905701 10817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Leader election won for term 8
I20260928 02:04:47.906064 10817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 LEADER]: Becoming Leader. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Running, Role: LEADER
I20260928 02:04:47.906605 10817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 6.21, Last appended by leader: 21, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:47.909809 10820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 8 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:47.910102 10820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:47.910730 10822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:47.913092 10822 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:47.913542 10822 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:47.913901 10822 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:47.916061 10822 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:47.916473 10822 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:47.916764 10822 tablet_loader.cc:96] loaded metadata for tablet dbff92b3afb64048bb1dda2517c82f7a (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:47.917318 10822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:47.918361 10822 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:47.918435 10822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:47.919552 10822 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:47.920480 10822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Loaded TSK: 0
I20260928 02:04:47.921371 10822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:47.922479 10822 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:47.923570 10822 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:47.925558 10556 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:47.925717 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:47.986472 10749 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58506:
table { table_name: "default.Table_b2462a042ad34570ac6321c1d9e4212c" } new_table_name: "default.Table_b6652c20a8c44c2d8e1ec6246b2159a9" modify_external_catalogs: true
I20260928 02:04:47.987643 10743 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58506:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
I20260928 02:04:47.986506 10750 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_7f2d941f68d647be9c91a40928667e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:47.986866 10746 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:47.987277 10748 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:47.987007 10747 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:47.986986 10745 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:47.988670 10741 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:47.987757 10744 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:47.989486 10742 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58506:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260928 02:04:47.992228 10802 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7fab4572d9ad google::LogMessage::Fail() at ??:0
@ 0x7fab4573088a google::LogMessage::SendToLog() at ??:0
@ 0x7fab4572d4de google::LogMessage::Flush() at ??:0
@ 0x7fab4572d6d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7fab5b2cfba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fab5b2ce901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fab5b2d566c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fab5b2d54e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fab48357eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fab4b6316db start_thread at ??:0
@ 0x7fab445c761f clone at ??:0
W20260928 02:04:48.185089 9699 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:48.186414 9818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.189:36913)
W20260928 02:04:48.186894 9818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:48.187505 9818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.189:36913)
I20260928 02:04:48.773185 10841 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:48.773427 10841 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:48.774072 10841 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:48.775403 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260928 02:04:48.776739 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:48.776993 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:48.777099 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:48.777318 10841 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260928 02:04:49.898495 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--webserver_interface=127.8.174.189
--webserver_port=34795
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260928 02:04:50.235293 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:50.235601 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:50.235687 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:50.243352 10848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:50.243481 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:50.243544 10848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:50.243620 10848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:50.254105 10848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.174.189
--webserver_port=34795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:50.255628 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:50.257447 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:50.270795 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:50.270820 10854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:50.271027 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:50.271869 10848 server_base.cc:1034] running on GCE node
I20260928 02:04:50.272722 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:50.274329 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:50.275629 10848 hybrid_clock.cc:648] HybridClock initialized: now 1790561090275537 us; error 53 us; skew 500 ppm
I20260928 02:04:50.276156 10848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:50.278999 10848 webserver.cc:460] Webserver started at http://127.8.174.189:34795/ using document root <none> and password file <none>
I20260928 02:04:50.279826 10848 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:50.279945 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:50.286021 10848 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:04:50.289373 10862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:50.291049 10848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:50.291220 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:50.292229 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:50.307588 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:50.308528 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:50.308817 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:50.327934 10848 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.189:36913
I20260928 02:04:50.327991 10913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.189:36913 every 8 connection(s)
I20260928 02:04:50.329486 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
I20260928 02:04:50.336903 10914 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:50.336889 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 10848
I20260928 02:04:50.341631 10914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap starting.
I20260928 02:04:50.352687 10914 log.cc:824] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:50.373947 10914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=21 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260928 02:04:50.374636 10914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap complete.
I20260928 02:04:50.382828 10914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:50.384580 10914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Initialized, Role: FOLLOWER
I20260928 02:04:50.385371 10914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 8.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:50.386286 10918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:50.386739 10918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:50.387671 10914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:50.414238 10932 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:50.414455 10932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: loading cluster ID for follower catalog manager: success
I20260928 02:04:50.419831 10932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: acquiring CA information for follower catalog manager: success
I20260928 02:04:50.422966 10932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:50.438634 9540 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:50.435992 10848 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:50.443173 10848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:50.444484 10879 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:38173; Asking this server to re-register.
I20260928 02:04:50.445413 9540 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:50.447413 10879 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:50.643321 10938 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:50.643559 10938 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:50.644326 10938 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:50.647632 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:50.652710 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" candidate_term: 9 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "5c9432f318fa44279928ebd2043a6640" is_pre_election: true
I20260928 02:04:50.653421 10889 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f09e70c2f7354c4e888b5aa6f0175a22 for term 9 because replica has last-logged OpId of term: 8 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 17.
I20260928 02:04:50.654125 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:50.654538 10938 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260928 02:04:50.697310 9812 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:50.698761 10879 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:47871; Asking this server to re-register.
I20260928 02:04:50.699532 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:50.701016 10879 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:50.741807 9677 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:50.743309 10879 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:60379; Asking this server to re-register.
I20260928 02:04:50.744029 9677 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:50.745395 10879 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:50.938691 10943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:50.938982 10943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:50.940634 10943 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:50.941249 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 9 candidate_status { last_received { term: 8 index: 22 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:50.941565 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 8.
W20260928 02:04:50.941571 10863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:50.942144 10866 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters:
I20260928 02:04:50.942548 10943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260928 02:04:50.942690 10943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:50.942795 10943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 8 FOLLOWER]: Advancing to term 9
W20260928 02:04:50.943123 10863 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:50.947013 10943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:50.947641 10943 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 9 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:50.948257 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 9 candidate_status { last_received { term: 8 index: 22 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
I20260928 02:04:50.948470 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 8 FOLLOWER]: Advancing to term 9
W20260928 02:04:50.949025 10863 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:50.950930 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 9.
I20260928 02:04:50.951324 10866 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 7017d177043645b5afe32c476055db7e
I20260928 02:04:50.951611 10943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Leader election won for term 9
I20260928 02:04:50.952092 10943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 LEADER]: Becoming Leader. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Running, Role: LEADER
I20260928 02:04:50.952551 10943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 8.22, Last appended by leader: 22, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:50.954974 10946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 9 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:50.955245 10946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:50.955819 10948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:50.958884 10948 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:50.959369 10948 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:50.959671 10948 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:50.961678 10948 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:50.961994 10948 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:50.962348 10948 tablet_loader.cc:96] loaded metadata for tablet dbff92b3afb64048bb1dda2517c82f7a (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:50.962867 10948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:50.963848 10948 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:50.963918 10948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:50.965375 10948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:50.966250 10948 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Loaded TSK: 0
I20260928 02:04:50.967270 10948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:50.968096 10948 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:50.968902 10948 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:51.044359 10879 master_service.cc:946] ReplaceTablet: received request to replace tablet e65189e85b2641098a1bdfca68c049c4 from {username='slave'} at 127.0.0.1:59116
I20260928 02:04:51.044428 10878 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:59116
I20260928 02:04:51.045569 10872 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59116:
table { table_name: "default.Table_b2462a042ad34570ac6321c1d9e4212c" } new_table_name: "default.Table_b6652c20a8c44c2d8e1ec6246b2159a9" modify_external_catalogs: true
I20260928 02:04:51.045308 10877 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:51.045989 10878 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:51.046440 10879 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:51.046773 10871 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:51.046916 10870 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59116:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
I20260928 02:04:51.047112 10876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:51.047380 10874 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:51.047536 10873 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_7f2d941f68d647be9c91a40928667e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:51.047868 10875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59116:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260928 02:04:51.052412 10931 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7f6d31e079ad google::LogMessage::Fail() at ??:0
@ 0x7f6d31e0a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6d31e074de google::LogMessage::Flush() at ??:0
@ 0x7f6d31e076d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7f6d479a9ba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f6d479a8901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f6d479af66c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f6d479af4e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6d34a31eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6d37d0b6db start_thread at ??:0
@ 0x7f6d30ca161f clone at ??:0
W20260928 02:04:51.257148 9699 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:51.257148 9564 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:51.257748 9818 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:51.258481 9818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.189:36913)
W20260928 02:04:51.259155 9818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:51.259779 9818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.189:36913)
I20260928 02:04:51.938867 10965 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:04:51.939085 10965 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:51.939668 10965 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:51.940639 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:04:51.940812 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:51.940950 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:04:51.941175 10965 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:04:52.928716 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:04:52.928944 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:53.243085 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--webserver_interface=127.8.174.189
--webserver_port=34795
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260928 02:04:53.588771 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:53.589097 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:53.589214 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:53.597301 10969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:53.597501 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:53.597602 10969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:53.597695 10969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:53.607774 10969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.174.189
--webserver_port=34795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:53.609164 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:53.611109 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:53.623425 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:53.623508 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:53.623416 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:53.624467 10969 server_base.cc:1034] running on GCE node
I20260928 02:04:53.625324 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:53.627061 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:53.628338 10969 hybrid_clock.cc:648] HybridClock initialized: now 1790561093628241 us; error 31 us; skew 500 ppm
I20260928 02:04:53.628731 10969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:53.631166 10969 webserver.cc:460] Webserver started at http://127.8.174.189:34795/ using document root <none> and password file <none>
I20260928 02:04:53.631899 10969 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:53.632009 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:53.637692 10969 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:04:53.641206 10984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:53.642948 10969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 02:04:53.643162 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:53.644184 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:53.680330 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:53.681305 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:53.681602 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:53.699601 10969 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.189:36913
I20260928 02:04:53.699620 11035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.189:36913 every 8 connection(s)
I20260928 02:04:53.701138 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
I20260928 02:04:53.702039 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 10969
I20260928 02:04:53.708971 11036 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:53.713426 11036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap starting.
I20260928 02:04:53.723508 11036 log.cc:824] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:53.744222 11036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=21 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260928 02:04:53.744951 11036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap complete.
I20260928 02:04:53.752779 11036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:53.754213 11036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Initialized, Role: FOLLOWER
I20260928 02:04:53.754839 11036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 9.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:53.755609 11040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:53.755931 11040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:53.756722 11036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:53.774984 9812 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:53.780740 11001 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:56841; Asking this server to re-register.
I20260928 02:04:53.782819 9677 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:53.783427 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:53.784107 11054 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:53.784194 11054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: loading cluster ID for follower catalog manager: success
I20260928 02:04:53.785764 11000 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:53.788694 11000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:54813; Asking this server to re-register.
I20260928 02:04:53.789335 9677 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:53.789603 11054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: acquiring CA information for follower catalog manager: success
I20260928 02:04:53.790843 11000 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:53.793852 11054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:53.806556 10969 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:53.812757 10969 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:53.965523 9540 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:53.967089 11000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:38139; Asking this server to re-register.
I20260928 02:04:53.967691 9540 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:53.969069 11000 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:54.217872 10970 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:54.218116 10970 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:54.218888 10970 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:54.219920 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:04:54.220120 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:04:54.220216 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:54.220444 10970 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:04:54.290172 11062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:54.290489 11062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:54.292708 11062 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:54.293160 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
I20260928 02:04:54.293421 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 9.
W20260928 02:04:54.293710 10985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:54.293998 10988 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters:
I20260928 02:04:54.294411 11062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260928 02:04:54.294591 11062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:54.295161 11062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 9 FOLLOWER]: Advancing to term 10
W20260928 02:04:54.295249 10985 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:54.299254 11062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:54.299897 11062 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 10 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:54.300375 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
I20260928 02:04:54.300629 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 9 FOLLOWER]: Advancing to term 10
W20260928 02:04:54.301182 10985 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:54.302704 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 10.
I20260928 02:04:54.303117 10988 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 7017d177043645b5afe32c476055db7e
I20260928 02:04:54.303380 11062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Leader election won for term 10
I20260928 02:04:54.303781 11062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 LEADER]: Becoming Leader. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Running, Role: LEADER
I20260928 02:04:54.304234 11062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 9.23, Last appended by leader: 23, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:54.306841 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 10 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:54.307193 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:54.307778 11066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:54.310122 11066 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:54.310636 11066 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:54.310958 11066 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:54.312901 11066 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:54.313180 11066 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:54.313459 11066 tablet_loader.cc:96] loaded metadata for tablet dbff92b3afb64048bb1dda2517c82f7a (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:54.313952 11066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:54.314903 11066 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:54.314972 11066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:54.316155 11066 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:54.316993 11066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Loaded TSK: 0
I20260928 02:04:54.317867 11066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:54.318732 11066 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:54.319491 11066 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:54.345927 11000 master_service.cc:946] ReplaceTablet: received request to replace tablet e65189e85b2641098a1bdfca68c049c4 from {username='slave'} at 127.0.0.1:58874
I20260928 02:04:54.346088 11001 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:58874
I20260928 02:04:54.346221 10999 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:58874
I20260928 02:04:54.346546 10998 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:58874
I20260928 02:04:54.348441 11001 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58874:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
I20260928 02:04:54.348922 11000 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58874:
table { table_name: "default.Table_b2462a042ad34570ac6321c1d9e4212c" } new_table_name: "default.Table_b6652c20a8c44c2d8e1ec6246b2159a9" modify_external_catalogs: true
I20260928 02:04:54.347313 10997 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:54.347216 10996 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:54.349555 10995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:54.349560 10999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:54.349731 10992 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:54.347394 10993 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_7f2d941f68d647be9c91a40928667e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:54.350236 10994 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:54.350809 10998 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58874:
name: "default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260928 02:04:54.355290 11053 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7f11303dc9ad google::LogMessage::Fail() at ??:0
@ 0x7f11303df88a google::LogMessage::SendToLog() at ??:0
@ 0x7f11303dc4de google::LogMessage::Flush() at ??:0
@ 0x7f11303dc6d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7f1145f7eba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f1145f7d901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f1145f8466c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f1145f844e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1133006eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f11362e06db start_thread at ??:0
@ 0x7f112f27661f clone at ??:0
W20260928 02:04:54.555307 9427 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:54.555876 9818 connection.cc:535] client connection to 127.8.174.189:36913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:04:54.556633 9818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.189:36913)
W20260928 02:04:54.557317 9818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:54.557973 9818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.189:36913)
I20260928 02:04:56.257287 11084 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:56.257526 11084 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:04:56.258278 11084 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:56.259413 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:04:56.259591 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:04:56.259704 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:04:56.259970 11084 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:04:56.482491 11086 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:04:56.483007 11086 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:04:56.483894 11086 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:04:56.485213 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:04:56.485412 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:04:56.485525 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:04:56.485769 11086 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:04:57.019295 8890 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
/tmp/dist-test-taskGCZ99a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--webserver_interface=127.8.174.189
--webserver_port=34795
--builtin_ntp_servers=127.8.174.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260928 02:04:57.363946 11088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:57.364295 11088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:57.364415 11088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:57.372036 11088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:57.372152 11088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:57.372193 11088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:57.372249 11088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:57.381919 11088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.174.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39769
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.174.189:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.174.189
--webserver_port=34795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:57.383500 11088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:57.385856 11088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:57.397943 11094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:57.397953 11095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:57.398206 11097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:57.398715 11088 server_base.cc:1034] running on GCE node
I20260928 02:04:57.399544 11088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:57.401022 11088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:57.402256 11088 hybrid_clock.cc:648] HybridClock initialized: now 1790561097402191 us; error 32 us; skew 500 ppm
I20260928 02:04:57.402660 11088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:57.405254 11088 webserver.cc:460] Webserver started at http://127.8.174.189:34795/ using document root <none> and password file <none>
I20260928 02:04:57.405979 11088 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:57.406088 11088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:57.411873 11088 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:04:57.415346 11103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:57.417078 11088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:04:57.417270 11088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data,/tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
uuid: "5c9432f318fa44279928ebd2043a6640"
format_stamp: "Formatted at 2026-09-28 02:04:25 on dist-test-slave-3l00"
I20260928 02:04:57.418385 11088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:57.433696 11088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:57.434631 11088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:57.434914 11088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:57.453197 11088 rpc_server.cc:307] RPC server started. Bound to: 127.8.174.189:36913
I20260928 02:04:57.453204 11154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.174.189:36913 every 8 connection(s)
I20260928 02:04:57.454772 11088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGCZ99a/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1790561048912601-8890-0/minicluster-data/master-1/data/info.pb
I20260928 02:04:57.459440 8890 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGCZ99a/build/asan/bin/kudu as pid 11088
I20260928 02:04:57.462265 11155 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:04:57.466603 11155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap starting.
I20260928 02:04:57.477197 11155 log.cc:824] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:57.496572 11155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=21 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260928 02:04:57.497184 11155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Bootstrap complete.
I20260928 02:04:57.504834 11155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:57.506507 11155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Initialized, Role: FOLLOWER
I20260928 02:04:57.507302 11155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 10.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:57.508142 11159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:57.508569 11159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:04:57.509443 11155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:57.521337 11173 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:57.521528 11173 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: loading cluster ID for follower catalog manager: success
I20260928 02:04:57.526342 11173 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: acquiring CA information for follower catalog manager: success
I20260928 02:04:57.529069 11173 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:04:57.556763 11088 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:57.562978 11088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:57.570842 9540 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:57.574376 11120 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" instance_seqno: 1790561066552913) as {username='slave'} at 127.8.174.129:36791; Asking this server to re-register.
I20260928 02:04:57.575150 9540 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:57.577030 11120 ts_manager.cc:194] Registered new tserver with Master: ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359)
I20260928 02:04:57.804181 9812 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:57.805506 11120 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c51770d4a61463c9330530e0d019d98" instance_seqno: 1790561067515011) as {username='slave'} at 127.8.174.131:38881; Asking this server to re-register.
I20260928 02:04:57.806111 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:57.808099 9677 heartbeater.cc:347] Connected to a master server at 127.8.174.189:36913
I20260928 02:04:57.808238 11120 ts_manager.cc:194] Registered new tserver with Master: 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
I20260928 02:04:57.809398 11120 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53ad00df981d4266a118559f648b1abe" instance_seqno: 1790561067039631) as {username='slave'} at 127.8.174.130:42723; Asking this server to re-register.
I20260928 02:04:57.809942 9677 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:57.811221 11120 ts_manager.cc:194] Registered new tserver with Master: 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843)
I20260928 02:04:57.940857 10556 master_runner.cc:186] Error getting master registration for 127.8.174.188:44075: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111),
I20260928 02:04:57.941069 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:04:58.002530 11159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:58.002794 11159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:58.004601 11159 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:58.005213 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 11 candidate_status { last_received { term: 10 index: 24 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" is_pre_election: true
W20260928 02:04:58.005367 11104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:58.005532 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 10.
I20260928 02:04:58.006108 11107 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters:
I20260928 02:04:58.006538 11159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260928 02:04:58.006682 11159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:58.006788 11159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 10 FOLLOWER]: Advancing to term 11
W20260928 02:04:58.006982 11104 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:58.011269 11159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:58.011920 11159 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 11 election: Requested vote from peers f09e70c2f7354c4e888b5aa6f0175a22 (127.8.174.190:39911), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
I20260928 02:04:58.012420 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9432f318fa44279928ebd2043a6640" candidate_term: 11 candidate_status { last_received { term: 10 index: 24 } } ignore_live_leader: false dest_uuid: "f09e70c2f7354c4e888b5aa6f0175a22"
I20260928 02:04:58.012677 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 10 FOLLOWER]: Advancing to term 11
W20260928 02:04:58.013341 11104 leader_election.cc:334] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:04:58.014859 10609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9432f318fa44279928ebd2043a6640 in term 11.
I20260928 02:04:58.015283 11107 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9432f318fa44279928ebd2043a6640, f09e70c2f7354c4e888b5aa6f0175a22; no voters: 7017d177043645b5afe32c476055db7e
I20260928 02:04:58.015568 11159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 11 FOLLOWER]: Leader election won for term 11
I20260928 02:04:58.016085 11159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [term 11 LEADER]: Becoming Leader. State: Replica: 5c9432f318fa44279928ebd2043a6640, State: Running, Role: LEADER
I20260928 02:04:58.016559 11159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 10.24, Last appended by leader: 24, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:58.018983 11181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9432f318fa44279928ebd2043a6640. Latest consensus state: current_term: 11 leader_uuid: "5c9432f318fa44279928ebd2043a6640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } } }
I20260928 02:04:58.019197 11181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:58.019690 11183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:58.022048 11183 catalog_manager.cc:669] Loaded metadata for table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7]
I20260928 02:04:58.022604 11183 catalog_manager.cc:669] Loaded metadata for table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3]
I20260928 02:04:58.022959 11183 catalog_manager.cc:669] Loaded metadata for table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898]
I20260928 02:04:58.024945 11183 tablet_loader.cc:96] loaded metadata for tablet 4d4dc773cf234b2aa21013ec12b52290 (table default.table_b2462a042ad34570ac6321c1d9e4212c [id=29b0aadc8be8433aaf9723eced4d28f7])
I20260928 02:04:58.025226 11183 tablet_loader.cc:96] loaded metadata for tablet 54a8ea3c4dee42d09d5938e9202df82f (table default.table_cb325c0bff7b4d7ca343532d7e14b25f [id=4411fe865a5f4e8187032edebc45b3d3])
I20260928 02:04:58.025494 11183 tablet_loader.cc:96] loaded metadata for tablet dbff92b3afb64048bb1dda2517c82f7a (table default.table_ccd4aeece7af4142a6e4c84a9bc39b89 [id=972b59b79c6b471a82a92801303d3898])
I20260928 02:04:58.026026 11183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:58.027086 11183 catalog_manager.cc:1259] Loaded cluster ID: efac844546f04b3f9eb8b528b12fd138
I20260928 02:04:58.027172 11183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:58.028450 11183 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:58.029309 11183 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c9432f318fa44279928ebd2043a6640: Loaded TSK: 0
I20260928 02:04:58.030243 11183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:58.031201 11183 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260928 02:04:58.031967 11183 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260928 02:04:58.048877 11120 master_service.cc:946] ReplaceTablet: received request to replace tablet e65189e85b2641098a1bdfca68c049c4 from {username='slave'} at 127.0.0.1:59418
I20260928 02:04:58.049077 11119 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:59418
I20260928 02:04:58.049402 11117 master_service.cc:946] ReplaceTablet: received request to replace tablet e02df1dfaba348449be5816040ea63bd from {username='slave'} at 127.0.0.1:59418
I20260928 02:04:58.049468 11116 master_service.cc:946] ReplaceTablet: received request to replace tablet e65189e85b2641098a1bdfca68c049c4 from {username='slave'} at 127.0.0.1:59418
I20260928 02:04:58.049861 11118 master_service.cc:946] ReplaceTablet: received request to replace tablet 006d1c27cc874f11bd66fb947942a96a from {username='slave'} at 127.0.0.1:59418
I20260928 02:04:58.050894 11117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_31ff78a15f5242599f77f5785ac971f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.051218 11113 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_ae2bdbd94c92423aaea96f97b87e34ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.051595 11120 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59418:
table { table_name: "default.Table_b2462a042ad34570ac6321c1d9e4212c" } new_table_name: "default.Table_b6652c20a8c44c2d8e1ec6246b2159a9" modify_external_catalogs: true
I20260928 02:04:58.052062 11111 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_406b409ab888471581f1119b16f99179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.052194 11112 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_526955b93b8245538db639f0cced29ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.050901 11114 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_21e7ecfcfe254c049a51ef89351c6a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.052798 11116 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_8d14d0f90c2b4abea725b02c59da2573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.053290 11118 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_7f2d941f68d647be9c91a40928667e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.053650 11115 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59418:
name: "default.Table_657e2574ee3d4050ab7aade380a35acc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:58.054841 11119 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59418:
table { table_name: "default.Table_cb325c0bff7b4d7ca343532d7e14b25f" } new_table_name: "default.Table_51f057b86c1d412d942febdeb10a0977" modify_external_catalogs: true
F20260928 02:04:58.056799 11172 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_b2462a042ad34570ac6321c1d9e4212c
*** Check failure stack trace: ***
@ 0x7f35235379ad google::LogMessage::Fail() at ??:0
@ 0x7f352353a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f35235374de google::LogMessage::Flush() at ??:0
@ 0x7f35235376d1 google::LogMessage::~LogMessage() at ??:0
@ 0x7f35390d9ba8 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f35390d8901 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f35390df66c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f35390df4e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3526161eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f352943b6db start_thread at ??:0
@ 0x7f35223d161f clone at ??:0
W20260928 02:04:58.253789 9818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:58.254194 9818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.189:36913)
W20260928 02:04:58.258045 9818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.189:36913)
I20260928 02:04:58.951002 11199 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:58.951206 11199 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:04:58.952065 11199 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:04:58.954473 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20260928 02:04:58.955917 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:04:58.956126 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:04:58.956264 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:04:58.956560 11199 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:00.017939 8890 master-stress-test.cc:578] Killing test thread 1/25
I20260928 02:05:01.940394 11211 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:05:01.940649 11211 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:01.941458 11211 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:01.942553 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:05:01.942729 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:05:01.942914 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:05:01.943177 11211 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:05:02.545568 11213 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:02.545806 11213 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:05:02.546821 11213 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:02.548266 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:05:02.548502 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:05:02.548615 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:02.549098 11213 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:02.944929 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:02.945120 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:05.600368 11221 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:05.600569 11221 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:05.601236 11221 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:05.602409 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:05:05.603952 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:05.604173 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:05.604302 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:05.604547 11221 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:05.959488 11220 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:05.959678 11220 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:05.960279 11220 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:05.961254 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:05:05.961427 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:05:05.961537 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:05.961755 11220 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:06.633370 11227 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:05:06.633641 11227 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:06.634615 11227 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:06.635885 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:05:06.636101 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:05:06.636217 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:05:06.636494 11227 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:05:07.948293 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:07.948496 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:09.398972 11234 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:09.399180 11234 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:09.399833 11234 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:09.402196 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:05:09.402494 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:05:09.402621 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:09.402940 11234 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:11.474509 11240 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:11.474756 11240 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:05:11.475489 11240 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:11.476559 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:05:11.476732 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:05:11.476850 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:11.477106 11240 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:12.952749 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:12.952986 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:13.300154 11247 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:05:13.300421 11247 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:13.301429 11247 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:13.302582 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:05:13.302779 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:05:13.303051 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:05:13.303349 11247 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:05:17.956375 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:17.956571 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:18.765537 11259 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:18.765735 11259 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:18.766441 11259 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:18.767457 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260928 02:05:18.768863 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:18.769089 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:18.769234 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:18.769532 11259 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:20.453523 11264 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:20.453752 11264 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:20.454658 11264 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:20.455684 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:05:20.455849 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:05:20.455965 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:20.456221 11264 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:22.959731 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:22.959980 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:25.015190 11275 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:25.015424 11275 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:25.016109 11275 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:25.017421 10583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:05:25.019042 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:25.019268 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:25.019390 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:25.019646 11275 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:25.874714 11274 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:25.874905 11274 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:05:25.875547 11274 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:25.876431 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:05:25.876611 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:05:25.876720 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:25.876999 11274 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:27.963202 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:27.963389 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:05:28.151686 9810 heartbeater.cc:644] Failed to heartbeat to 127.8.174.188:44075 (85 consecutive failures): Network error: Failed to ping master at 127.8.174.188:44075: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:05:28.333638 9540 heartbeater.cc:644] Failed to heartbeat to 127.8.174.189:36913 (60 consecutive failures): Network error: Failed to ping master at 127.8.174.189:36913: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:28.378177 9675 heartbeater.cc:644] Failed to heartbeat to 127.8.174.188:44075 (86 consecutive failures): Network error: Failed to ping master at 127.8.174.188:44075: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:29.174014 11287 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:05:29.174237 11287 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:29.175194 11287 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:29.176106 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:05:29.176270 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:05:29.176386 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:05:29.176640 11287 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:05:32.764308 11295 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:05:32.764530 11295 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:32.765264 11295 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:32.766247 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:05:32.766459 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:05:32.766564 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:05:32.766770 11295 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:05:32.966529 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:32.966701 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:34.110948 11300 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:34.111184 11300 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:05:34.111979 11300 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:34.113301 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:05:34.113485 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:05:34.113605 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:34.113930 11300 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:36.202191 11304 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:36.202457 11304 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:36.203200 11304 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:36.204329 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:05:36.205614 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:36.205760 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:36.205909 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:36.206255 11304 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:36.714870 11309 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:36.715122 11309 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:36.715996 11309 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:36.716929 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:05:36.717077 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:05:36.717271 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:36.717502 11309 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:37.275696 11312 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:37.275957 11312 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:37.276629 11312 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:37.278661 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:37.278890 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:37.279017 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:37.279224 11312 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:37.969929 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:37.970122 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:40.473202 11324 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:40.473433 11324 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:40.474211 11324 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:40.477263 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:40.477463 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:40.477619 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:40.478000 11324 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:42.974118 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:42.974408 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:45.024186 11340 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:45.024400 11340 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:45.025167 11340 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:45.026335 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:05:45.026535 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:05:45.026647 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:45.026937 11340 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:45.192903 11340 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:45.193202 11340 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:05:45.194046 11340 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:45.195209 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:05:45.195407 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:05:45.195541 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:45.195782 11340 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:45.345551 11342 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:05:45.345793 11342 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:45.346784 11342 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:45.348215 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:05:45.348474 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:05:45.348656 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:05:45.349025 11342 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:05:46.670452 11345 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:46.670768 11345 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:46.671612 11345 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:46.672966 10583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260928 02:05:46.674476 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:05:46.674693 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:05:46.674899 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:46.675220 11345 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:47.978331 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:47.978562 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:52.600323 11368 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:05:52.600581 11368 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:05:52.601372 11368 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:52.602321 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:05:52.602505 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:05:52.602655 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:05:52.602917 11368 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:05:52.982151 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:52.982389 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:05:53.327634 11373 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:53.327966 11373 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:53.328742 11373 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:53.329452 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:05:53.331007 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:53.331210 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:53.331338 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:53.331574 11373 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:56.420526 11389 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:56.420792 11389 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:05:56.421615 11389 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:05:56.424495 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:05:56.424746 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:05:56.424916 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:05:56.425238 11389 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:05:57.696009 11402 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:57.696235 11402 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:05:57.697504 11402 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:05:57.698444 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:05:57.698604 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:05:57.698709 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:05:57.698907 11402 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:57.985013 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:05:57.985225 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:02.311457 11428 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:02.311709 11428 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:06:02.312397 11428 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:02.313373 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:06:02.313535 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:06:02.313683 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:02.313920 11428 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:02.991536 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:02.991708 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:04.788902 11449 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:04.789103 11449 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:04.789809 11449 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:04.790874 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260928 02:06:04.792208 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:06:04.792387 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:06:04.792514 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:04.792748 11449 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:05.998819 11461 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:05.999061 11461 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:05.999732 11461 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:06.001889 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:06:06.002118 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:06:06.002218 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:06.002467 11461 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:07.995021 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:07.995229 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:08.532466 11481 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:08.532706 11481 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:08.533528 11481 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:08.536787 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:06:08.537107 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:06:08.537274 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:08.537693 11481 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:10.663568 11497 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:10.663775 11497 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:10.664562 11497 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:10.665712 10583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260928 02:06:10.667191 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:06:10.667382 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:06:10.667508 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:10.667775 11497 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:11.606953 11507 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:06:11.607188 11507 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:06:11.608080 11507 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:11.608966 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:06:11.609140 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:06:11.609238 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:06:11.609493 11507 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:06:12.723316 11511 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:12.723565 11511 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:06:12.724449 11511 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:12.725433 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:06:12.725602 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:06:12.725729 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:12.726054 11511 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:12.998795 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:12.998962 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:18.001724 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:18.001868 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:21.491336 11568 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:21.491556 11568 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:06:21.492257 11568 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:21.493109 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:06:21.493256 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:06:21.493366 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:21.493597 11568 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:23.004818 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:23.005041 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:23.832257 11581 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:23.832520 11581 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:06:23.833237 11581 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:23.834192 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:06:23.834406 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:06:23.834584 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:23.834824 11581 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:25.481261 11593 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:06:25.481539 11593 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:06:25.482265 11593 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:25.483188 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:06:25.483359 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:06:25.483479 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:06:25.483716 11593 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:06:25.927263 11597 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:25.927481 11597 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:25.928323 11597 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:25.929415 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:06:25.930665 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:06:25.930838 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:06:25.930972 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:25.931190 11597 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:28.008880 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:28.009114 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:06:28.318511 9810 heartbeater.cc:644] Failed to heartbeat to 127.8.174.188:44075 (205 consecutive failures): Network error: Failed to ping master at 127.8.174.188:44075: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:06:28.503880 9540 heartbeater.cc:644] Failed to heartbeat to 127.8.174.189:36913 (180 consecutive failures): Network error: Failed to ping master at 127.8.174.189:36913: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:06:28.560761 9675 heartbeater.cc:644] Failed to heartbeat to 127.8.174.188:44075 (206 consecutive failures): Network error: Failed to ping master at 127.8.174.188:44075: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:06:33.012101 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:33.012291 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:35.063560 11656 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:35.063747 11656 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:06:35.064630 11656 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:35.065476 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:06:35.065673 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:06:35.065840 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:35.066110 11656 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:36.506348 11666 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:36.506577 11666 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:06:36.507234 11666 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:36.508100 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:06:36.508268 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:06:36.508355 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:36.508575 11666 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:38.015585 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:38.015772 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:43.006073 10588 maintenance_manager.cc:419] P f09e70c2f7354c4e888b5aa6f0175a22: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033369
I20260928 02:06:43.031647 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:43.031942 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:43.053617 10587 maintenance_manager.cc:643] P f09e70c2f7354c4e888b5aa6f0175a22: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.047s user 0.025s sys 0.005s Metrics: {"bytes_written":8272,"cfile_init":1,"dirs.queue_time_us":457,"dirs.run_cpu_time_us":385,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":12,"thread_start_us":580,"threads_started":2,"wal-append.queue_time_us":421}
I20260928 02:06:43.055284 10588 maintenance_manager.cc:419] P f09e70c2f7354c4e888b5aa6f0175a22: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 709 bytes on disk
I20260928 02:06:43.056171 10587 maintenance_manager.cc:643] P f09e70c2f7354c4e888b5aa6f0175a22: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20260928 02:06:43.695099 11715 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:06:43.695343 11715 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:06:43.696215 11715 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:43.697091 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:06:43.697269 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:06:43.697363 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:06:43.697585 11715 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:06:45.693681 11729 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:45.693914 11729 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:45.694869 11729 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:45.695969 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:06:45.697489 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:06:45.697664 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:06:45.697793 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:45.698035 11729 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:47.152035 11741 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:47.152240 11741 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:47.152884 11741 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:47.155081 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:06:47.155258 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:06:47.155412 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:47.155694 11741 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:48.034600 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:48.034762 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:48.851857 11755 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:48.852078 11755 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:06:48.852737 11755 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:48.853688 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:06:48.853859 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:06:48.853991 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:48.854215 11755 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:49.767805 11760 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:49.768060 11760 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:06:49.768668 11760 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:06:49.769610 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:06:49.769765 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:06:49.769893 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:06:49.770146 11760 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:53.038043 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:53.038261 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:06:55.554090 11793 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:55.554324 11793 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:06:55.555089 11793 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:06:55.556310 10583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260928 02:06:55.557715 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:06:55.557937 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:06:55.558037 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:06:55.558264 11793 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:06:58.041955 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:06:58.042163 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:01.873256 11834 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:01.873535 11834 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:07:01.874680 11834 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:01.875861 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:07:01.876047 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:07:01.876174 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:01.876426 11834 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:03.045265 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:03.045431 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:03.375638 11849 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:07:03.375916 11849 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:03.376988 11849 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:03.378067 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:07:03.378243 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:07:03.378382 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:07:03.378588 11849 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:07:04.625945 11857 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:04.626168 11857 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:04.627012 11857 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:04.628021 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:07:04.628201 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:07:04.628298 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:04.628530 11857 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:08.048815 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:08.048980 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:11.764402 11933 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:07:11.764660 11933 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:11.765413 11933 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:11.766453 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:07:11.766613 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:07:11.766794 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:07:11.767046 11933 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:07:13.052331 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:13.052502 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:14.565178 11963 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:14.565444 11963 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:07:14.566388 11963 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:07:14.567595 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:07:14.568959 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:07:14.569240 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:07:14.569399 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:07:14.569691 11963 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:07:18.055789 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:18.055951 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:21.031054 12032 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:21.031270 12032 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:07:21.031936 12032 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:07:21.033011 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:07:21.034134 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:07:21.034409 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:07:21.034546 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:07:21.034845 12032 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:07:21.545161 12038 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:21.545375 12038 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:07:21.546277 12038 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:21.547338 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:07:21.547505 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:07:21.547616 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:21.547902 12038 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:21.796885 12038 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:21.797080 12038 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:21.797753 12038 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:21.798861 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:07:21.799113 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:07:21.799238 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:21.799554 12038 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:23.059007 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:23.059176 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:24.408447 12070 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:07:24.408636 12070 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:24.409463 12070 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:24.410521 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:07:24.410709 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:07:24.410871 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:07:24.411125 12070 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:07:28.062376 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:28.062553 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:07:28.479923 9810 heartbeater.cc:644] Failed to heartbeat to 127.8.174.188:44075 (325 consecutive failures): Network error: Failed to ping master at 127.8.174.188:44075: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:07:28.659963 9540 heartbeater.cc:644] Failed to heartbeat to 127.8.174.189:36913 (300 consecutive failures): Network error: Failed to ping master at 127.8.174.189:36913: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:07:28.758028 9675 heartbeater.cc:644] Failed to heartbeat to 127.8.174.188:44075 (326 consecutive failures): Network error: Failed to ping master at 127.8.174.188:44075: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:07:31.303550 12141 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:07:31.303767 12141 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:31.304524 12141 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:31.305727 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:07:31.305902 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:07:31.306012 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:07:31.306288 12141 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:07:33.065531 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:33.065712 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:34.372642 12162 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:34.372867 12162 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:34.373579 12162 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:34.374529 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:07:34.374701 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:07:34.374915 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:34.375136 12162 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:36.130358 12162 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:36.130609 12162 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:07:36.131333 12162 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:36.132316 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:07:36.132500 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:07:36.132647 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:36.132937 12162 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:38.070473 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:38.070631 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:39.715672 12217 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:39.715987 12217 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:07:39.717038 12217 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:39.718338 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:07:39.718577 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:07:39.718725 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:39.718988 12217 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:39.963636 12222 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:39.963855 12222 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:07:39.964731 12222 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:07:39.966184 10583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:07:39.967944 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:07:39.968314 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:07:39.968508 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:07:39.968845 12222 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:07:42.407275 12249 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:42.407472 12249 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:42.408208 12249 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:42.409228 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:07:42.409449 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:07:42.409592 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:42.409858 12249 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:43.073477 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:43.073649 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:43.859148 12265 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:07:43.859352 12265 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:43.860205 12265 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:43.861292 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:07:43.861467 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:07:43.861627 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:07:43.861851 12265 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:07:48.077214 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:48.077545 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:48.095058 12306 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:48.095323 12306 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:07:48.096186 12306 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:07:48.096956 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:07:48.098449 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:07:48.098730 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:07:48.098851 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:07:48.099236 12306 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:07:48.514019 12313 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:48.514258 12313 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:48.515002 12313 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:48.516076 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:07:48.516255 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:07:48.516388 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:48.516659 12313 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:49.280006 12319 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:49.280294 12319 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:07:49.281160 12319 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:07:49.283527 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:07:49.283803 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:07:49.283917 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:07:49.284197 12319 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:07:52.743557 12355 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:52.743806 12355 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:07:52.744513 12355 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:52.745437 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:07:52.745648 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:07:52.745790 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:07:52.746034 12355 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:07:53.080523 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:53.080734 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:55.002524 12379 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:07:55.002818 12379 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:07:55.003544 12379 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:07:55.004807 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:07:55.005082 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:07:55.005244 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:07:55.005532 12379 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:07:58.084647 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:07:58.084888 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:01.671129 12435 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:01.671352 12435 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:01.672290 12435 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:01.673619 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:08:01.673839 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:08:01.673993 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:01.674232 12435 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:03.088075 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:03.088280 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:06.592480 12481 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:06.592787 12481 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:08:06.594835 12481 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:08:06.595166 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260928 02:08:06.597119 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:08:06.597373 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:08:06.597548 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:08:06.597834 12481 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:08:08.091089 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:08.091254 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:11.573316 12533 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:11.573513 12533 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:08:11.574251 12533 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:08:11.576745 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:08:11.576907 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:08:11.577024 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:08:11.577261 12533 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:08:12.492343 12543 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:12.492558 12543 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:08:12.493227 12543 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:12.494112 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:08:12.494284 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:08:12.494452 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:12.494750 12543 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:13.094645 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:13.094818 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:13.734962 12560 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:08:13.735185 12560 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:13.736050 12560 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:13.737051 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:08:13.737229 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:08:13.737327 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:08:13.737560 12560 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:08:18.097849 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:18.098012 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:18.616971 12605 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:18.617200 12605 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:08:18.617882 12605 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:08:18.618927 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260928 02:08:18.620234 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:08:18.620478 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:08:18.620579 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:08:18.620836 12605 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:08:21.137310 12632 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:21.137501 12632 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:21.138422 12632 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:21.139310 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:08:21.139485 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:08:21.139613 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:21.139843 12632 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:23.100930 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:23.101083 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:23.500232 12642 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:23.500433 12642 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:08:23.501206 12642 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:23.502041 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:08:23.502185 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:08:23.502377 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:23.502622 12642 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:24.084251 12659 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:08:24.084493 12659 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:24.085276 12659 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:24.086220 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:08:24.086416 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:08:24.086568 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:08:24.086819 12659 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:08:28.104203 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:28.104449 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:08:28.642402 9810 heartbeater.cc:644] Failed to heartbeat to 127.8.174.188:44075 (445 consecutive failures): Network error: Failed to ping master at 127.8.174.188:44075: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:08:28.822062 9540 heartbeater.cc:644] Failed to heartbeat to 127.8.174.189:36913 (420 consecutive failures): Network error: Failed to ping master at 127.8.174.189:36913: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:08:28.965411 9677 heartbeater.cc:644] Failed to heartbeat to 127.8.174.189:36913 (420 consecutive failures): Network error: Failed to ping master at 127.8.174.189:36913: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:08:29.383652 12707 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:08:29.383839 12707 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:29.384702 12707 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:29.385999 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:08:29.386206 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:08:29.386382 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:08:29.386670 12707 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:08:29.699797 12713 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:29.700070 12713 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:08:29.700774 12713 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:29.701998 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:08:29.702165 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:08:29.702422 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:29.702725 12713 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:30.193594 12716 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:30.195109 12716 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:08:30.195914 12716 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:08:30.197127 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:08:30.198566 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:08:30.198760 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:08:30.198889 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:08:30.199120 12716 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:08:32.411270 12741 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:32.411525 12741 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:08:32.412175 12741 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:08:32.414469 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:08:32.414616 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:08:32.414754 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:08:32.415068 12741 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:08:32.588532 12749 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:08:32.588804 12749 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:32.589612 12749 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:32.590575 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:08:32.590787 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:08:32.590909 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:08:32.591203 12749 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:08:33.107535 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:33.107700 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:37.894501 12791 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:37.894680 12791 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:37.895501 12791 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:37.896390 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:08:37.896548 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:08:37.896654 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:37.896863 12791 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:38.110745 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:38.110935 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:38.880944 12807 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:08:38.881119 12807 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:38.881858 12807 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:38.882846 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:08:38.883077 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:08:38.883277 9565 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:08:38.883528 12807 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:08:42.331868 12834 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:42.332087 12834 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:08:42.332800 12834 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:42.333949 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:08:42.334121 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:08:42.334224 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:42.334483 12834 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:43.113832 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:43.113997 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:48.117190 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:48.117376 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:48.187765 12893 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:48.187974 12893 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:08:48.188887 12893 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:08:48.189430 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260928 02:08:48.190639 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:08:48.190791 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:08:48.190904 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:08:48.191169 12893 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:08:51.913439 12927 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:51.913652 12927 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:51.914368 12927 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:51.915341 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:08:51.915496 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:08:51.915607 9426 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:51.915839 12927 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:53.120539 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:53.120720 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:08:57.555348 12978 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:08:57.555600 12978 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:57.556699 12978 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:57.557610 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:08:57.557809 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:08:57.557981 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:08:57.558223 12978 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:08:58.086377 12983 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:58.086587 12983 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:08:58.087468 12983 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:08:58.088385 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:08:58.088559 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:08:58.088682 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:08:58.088936 12983 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:08:58.123848 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:08:58.124007 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:09:01.340377 13013 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:01.340591 13013 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:09:01.341306 13013 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:09:01.342260 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:09:01.342444 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:09:01.342550 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:09:01.342788 13013 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:09:02.159265 13023 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:02.159512 13023 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:09:02.160471 13023 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:09:02.163338 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:09:02.165148 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:09:02.165331 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
I20260928 02:09:02.165434 10583 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:09:02.165671 13023 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:09:03.127264 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:09:03.127435 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:09:08.130411 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:09:08.130618 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:09:10.076557 13097 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:10.076773 13097 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:09:10.077545 13097 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:09:10.078476 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:09:10.078655 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:09:10.078800 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:09:10.079071 13097 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:09:11.144488 13105 raft_consensus.cc:482] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:11.144711 13105 raft_consensus.cc:504] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:09:11.145411 13105 leader_election.cc:288] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:09:11.146378 9426 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
W20260928 02:09:11.146607 9429 leader_election.cc:341] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 4d4dc773cf234b2aa21013ec12b52290
I20260928 02:09:11.146708 9429 leader_election.cc:302] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:09:11.146925 13105 raft_consensus.cc:2726] T 4d4dc773cf234b2aa21013ec12b52290 P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:09:13.133898 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:09:13.134342 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:09:14.512281 13139 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:14.512497 13139 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:09:14.513290 13139 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:09:14.514755 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:09:14.516002 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:09:14.516196 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:09:14.516289 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:09:14.516497 13139 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:09:14.737893 13147 raft_consensus.cc:482] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c51770d4a61463c9330530e0d019d98)
I20260928 02:09:14.738117 13147 raft_consensus.cc:504] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } } peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } }
I20260928 02:09:14.739009 13147 leader_election.cc:288] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:09:14.739985 9565 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer ebc8a50f334349b68dfbeb1ae9fc3635 (127.8.174.129:44359): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
W20260928 02:09:14.740134 9566 leader_election.cc:341] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: e65189e85b2641098a1bdfca68c049c4
I20260928 02:09:14.740276 9566 leader_election.cc:302] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53ad00df981d4266a118559f648b1abe; no voters: 8c51770d4a61463c9330530e0d019d98, ebc8a50f334349b68dfbeb1ae9fc3635
I20260928 02:09:14.740526 13147 raft_consensus.cc:2726] T e65189e85b2641098a1bdfca68c049c4 P 53ad00df981d4266a118559f648b1abe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260928 02:09:16.142012 13155 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:16.142261 13155 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:09:16.143210 13155 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:09:16.144551 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:09:16.144769 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:09:16.144937 9429 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:09:16.145232 13155 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:09:18.137450 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:09:18.137619 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:09:23.111776 13219 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:23.112002 13219 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09e70c2f7354c4e888b5aa6f0175a22" member_type: VOTER last_known_addr { host: "127.8.174.190" port: 39911 } } peers { permanent_uuid: "5c9432f318fa44279928ebd2043a6640" member_type: VOTER last_known_addr { host: "127.8.174.189" port: 36913 } } peers { permanent_uuid: "7017d177043645b5afe32c476055db7e" member_type: VOTER last_known_addr { host: "127.8.174.188" port: 44075 } }
I20260928 02:09:23.112739 13219 leader_election.cc:288] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913), 7017d177043645b5afe32c476055db7e (127.8.174.188:44075)
W20260928 02:09:23.113904 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260928 02:09:23.115165 10583 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 7017d177043645b5afe32c476055db7e (127.8.174.188:44075): Network error: Client connection negotiation failed: client connection to 127.8.174.188:44075: connect: Connection refused (error 111)
W20260928 02:09:23.115312 10584 leader_election.cc:334] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5c9432f318fa44279928ebd2043a6640 (127.8.174.189:36913): Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
I20260928 02:09:23.115448 10584 leader_election.cc:302] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09e70c2f7354c4e888b5aa6f0175a22; no voters: 5c9432f318fa44279928ebd2043a6640, 7017d177043645b5afe32c476055db7e
I20260928 02:09:23.115800 13219 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f09e70c2f7354c4e888b5aa6f0175a22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260928 02:09:23.140384 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:09:23.140568 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:09:26.774869 13256 raft_consensus.cc:482] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:26.775130 13256 raft_consensus.cc:504] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ad00df981d4266a118559f648b1abe" member_type: VOTER last_known_addr { host: "127.8.174.130" port: 39843 } } peers { permanent_uuid: "8c51770d4a61463c9330530e0d019d98" member_type: VOTER last_known_addr { host: "127.8.174.131" port: 37953 } } peers { permanent_uuid: "ebc8a50f334349b68dfbeb1ae9fc3635" member_type: VOTER last_known_addr { host: "127.8.174.129" port: 44359 } }
I20260928 02:09:26.775784 13256 leader_election.cc:288] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843), 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953)
W20260928 02:09:26.776909 9429 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c51770d4a61463c9330530e0d019d98 (127.8.174.131:37953): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
W20260928 02:09:26.777103 9426 leader_election.cc:341] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53ad00df981d4266a118559f648b1abe (127.8.174.130:39843): Not found: Tablet not found: 54a8ea3c4dee42d09d5938e9202df82f
I20260928 02:09:26.777222 9426 leader_election.cc:302] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc8a50f334349b68dfbeb1ae9fc3635; no voters: 53ad00df981d4266a118559f648b1abe, 8c51770d4a61463c9330530e0d019d98
I20260928 02:09:26.777467 13256 raft_consensus.cc:2726] T 54a8ea3c4dee42d09d5938e9202df82f P ebc8a50f334349b68dfbeb1ae9fc3635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:09:28.143473 10556 master_runner.cc:186] Error getting master registration for 127.8.174.189:36913: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111),
I20260928 02:09:28.143656 10556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:09:28.472357 9820 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:09:28.472636 9820 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
W20260928 02:09:28.472798 9820 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
F20260928 02:09:28.483829 9836 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1790561368 (unix time) try "date -d @1790561368" if you are using GNU date ***
W20260928 02:09:28.485833 9818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.174.189:36913)
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489025 9831 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20260928 02:09:28.489230 9827 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f2d941f68d647be9c91a40928667e1f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489025 9831 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20260928 02:09:28.489230 9827 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f2d941f68d647be9c91a40928667e1f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)Timed out: Error creating table default.Table_21e7ecfcfe254c049a51ef89351c6a38 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489025 9831 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20260928 02:09:28.489230 9827 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f2d941f68d647be9c91a40928667e1f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)Timed out: Error creating table default.Table_21e7ecfcfe254c049a51ef89351c6a38 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489614 9834 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_406b409ab888471581f1119b16f99179 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489025 9831 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20260928 02:09:28.489230 9827 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f2d941f68d647be9c91a40928667e1f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)Timed out: Error creating table default.Table_21e7ecfcfe254c049a51ef89351c6a38 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489614 9834 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_406b409ab888471581f1119b16f99179 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489710 9832 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8d14d0f90c2b4abea725b02c59da2573 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
W20260928 02:09:28.490285 9818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.174.189:36913)
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489025 9831 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20260928 02:09:28.489230 9827 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f2d941f68d647be9c91a40928667e1f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)Timed out: Error creating table default.Table_21e7ecfcfe254c049a51ef89351c6a38 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489614 9834 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_406b409ab888471581f1119b16f99179 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489710 9832 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8d14d0f90c2b4abea725b02c59da2573 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.490628 9825 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489025 9831 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20260928 02:09:28.489230 9827 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f2d941f68d647be9c91a40928667e1f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)Timed out: Error creating table default.Table_21e7ecfcfe254c049a51ef89351c6a38 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489614 9834 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_406b409ab888471581f1119b16f99179 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489710 9832 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8d14d0f90c2b4abea725b02c59da2573 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.490628 9825 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.490862 9826 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ae2bdbd94c92423aaea96f97b87e34ae on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
W20260928 02:09:28.491657 9818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.174.189:36913)
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000022ba) received by PID 8890 (TID 0x7f02aa5d9700) from PID 8890; stack trace: ***
F20260928 02:09:28.486274 9833 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_483ce76de8ea43dea7e133f72fa81aa1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.488984 9830 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_657e2574ee3d4050ab7aade380a35acc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489025 9831 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20260928 02:09:28.489230 9827 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f2d941f68d647be9c91a40928667e1f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)Timed out: Error creating table default.Table_21e7ecfcfe254c049a51ef89351c6a38 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489614 9834 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_406b409ab888471581f1119b16f99179 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.489710 9832 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8d14d0f90c2b4abea725b02c59da2573 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.490628 9825 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5bd8e8cf6f884d509a1c10fc5c7fe32a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.490862 9826 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ae2bdbd94c92423aaea96f97b87e34ae on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)F20260928 02:09:28.493172 9835 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.174.190:39911,127.8.174.189:36913,127.8.174.188:44075, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.174.189:36913: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f02bdc78a46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7f02c1dba980 (unknown) at ??:0
@ 0x7f02bafa6e87 gsignal at ??:0
@ 0x7f02bafa87f1 abort at ??:0
@ 0x7f02bdc6a9ad google::LogMessage::Fail() at ??:0
@ 0x7f02bdc6d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f02bdc6a4de google::LogMessage::Flush() at ??:0
@ 0x7f02bdc6ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x404bb0 kudu::MasterStressTest::AlterTableThread() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/master-stress-test.cc:329
@ 0x444b44 std::__invoke_impl<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:73
@ 0x4449fd std::__invoke<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
@ 0x4449ba _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
@ 0x44496e std::thread::_Invoker<>::operator()() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
@ 0x4446a9 std::thread::_State_impl<>::_M_run() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
@ 0x7f02c66fb8cf execute_native_thread_routine at ??:0
@ 0x7f02c1daf6db start_thread at ??:0
@ 0x7f02bb08961f clone at ??:0