Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251028 08:53:26.137276 4964 test_util.cc:276] Using random seed: 640140700
2025-10-28T08:53:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T08:53:26Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1hpXWl/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1hpXWl/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/security.properties
2025-10-28 08:53:31: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1hpXWl/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1hpXWl/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20251028 08:53:41.044880 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41357
--webserver_interface=127.4.217.62
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251028 08:53:41.329998 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:41.330242 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:41.330356 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:41.337342 5207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:53:41.337406 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:41.337457 5207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:53:41.337495 5207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:53:41.345932 5207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:41.347172 5207 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:41.348716 5207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:41.356254 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:41.356207 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:41.356572 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:41.356992 5207 server_base.cc:1047] running on GCE node
I20251028 08:53:41.357583 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:41.358820 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:41.360014 5207 hybrid_clock.cc:648] HybridClock initialized: now 1761641621359980 us; error 25 us; skew 500 ppm
I20251028 08:53:41.360318 5207 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:41.362242 5207 webserver.cc:492] Webserver started at http://127.4.217.62:46365/ using document root <none> and password file <none>
I20251028 08:53:41.362876 5207 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:41.362979 5207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:41.363304 5207 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 08:53:41.365262 5207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/instance:
uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:53:41.365983 5207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal/instance:
uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:53:41.370766 5207 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251028 08:53:41.373548 5221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:41.374601 5207 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 08:53:41.374745 5207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:53:41.374919 5207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:41.388031 5207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:41.388718 5207 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:41.388942 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:41.405992 5207 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:41357
I20251028 08:53:41.406025 5272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.62:41357 every 8 connection(s)
I20251028 08:53:41.407145 5207 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/info.pb
I20251028 08:53:41.409991 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 5207
I20251028 08:53:41.410282 4964 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal/instance
I20251028 08:53:41.411212 5273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:41.413331 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--webserver_interface=127.4.217.61
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251028 08:53:41.418555 5273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
I20251028 08:53:41.438486 5273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } has no permanent_uuid. Determining permanent_uuid...
W20251028 08:53:41.439438 5224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:53:41.440858 5273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:41.490113 5273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } attempt: 1
W20251028 08:53:41.491643 5273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:41.559834 5273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } attempt: 2
W20251028 08:53:41.561491 5273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:41.652706 5273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } attempt: 3
W20251028 08:53:41.654350 5273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:53:41.712574 5274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:41.712831 5274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:41.712944 5274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:41.720520 5274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:53:41.720604 5274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:41.720700 5274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:53:41.720767 5274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:53:41.729660 5274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:41.731025 5274 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:41.732606 5274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:41.739691 5282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:41.739758 5283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:41.740218 5285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:41.740825 5274 server_base.cc:1047] running on GCE node
I20251028 08:53:41.741451 5274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:41.742645 5274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:41.743839 5274 hybrid_clock.cc:648] HybridClock initialized: now 1761641621743802 us; error 40 us; skew 500 ppm
I20251028 08:53:41.744134 5274 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:41.746016 5274 webserver.cc:492] Webserver started at http://127.4.217.61:43565/ using document root <none> and password file <none>
I20251028 08:53:41.746652 5274 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:41.746745 5274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:41.747077 5274 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 08:53:41.749136 5274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/instance:
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:53:41.749890 5274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal/instance:
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:53:41.754477 5274 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251028 08:53:41.757436 5291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:41.758549 5274 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251028 08:53:41.758688 5274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:53:41.758870 5274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:41.794198 5274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:41.794894 5274 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:41.795127 5274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:41.797555 5273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } attempt: 4
W20251028 08:53:41.799319 5273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:41.814267 5274 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:33257
I20251028 08:53:41.814306 5342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.61:33257 every 8 connection(s)
I20251028 08:53:41.815512 5274 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
I20251028 08:53:41.817615 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 5274
I20251028 08:53:41.817808 4964 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal/instance
I20251028 08:53:41.819142 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--webserver_interface=127.4.217.60
--webserver_port=0
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251028 08:53:41.819799 5343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:41.826336 5343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
I20251028 08:53:41.837782 5343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } has no permanent_uuid. Determining permanent_uuid...
I20251028 08:53:41.846354 5343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
W20251028 08:53:41.847275 5292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:53:41.848707 5343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:53:41.897964 5343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } attempt: 1
W20251028 08:53:41.899397 5343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:53:41.967603 5343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } attempt: 2
W20251028 08:53:41.969314 5343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:53:42.060514 5343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } attempt: 3
W20251028 08:53:42.062450 5343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:53:42.098531 5273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } attempt: 5
I20251028 08:53:42.104594 5273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
W20251028 08:53:42.106202 5273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:53:42.123356 5344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:42.123628 5344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:42.123739 5344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:42.131701 5344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:53:42.131784 5344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:42.131887 5344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:53:42.131953 5344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:53:42.141036 5344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:42.142434 5344 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:42.144078 5344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:42.151147 5353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:42.151155 5352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:42.151865 5355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:42.153018 5344 server_base.cc:1047] running on GCE node
I20251028 08:53:42.153679 5344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:42.154826 5344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:42.156005 5344 hybrid_clock.cc:648] HybridClock initialized: now 1761641622155976 us; error 23 us; skew 500 ppm
I20251028 08:53:42.156297 5344 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:42.157387 5273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } attempt: 1
I20251028 08:53:42.158640 5344 webserver.cc:492] Webserver started at http://127.4.217.60:41817/ using document root <none> and password file <none>
W20251028 08:53:42.158903 5273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:53:42.159276 5344 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:42.159376 5344 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:42.159709 5344 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 08:53:42.161873 5344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/instance:
uuid: "82b026d06b454d65b27216e3002f29af"
format_stamp: "Formatted at 2025-10-28 08:53:42 on dist-test-slave-cmml"
I20251028 08:53:42.162596 5344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal/instance:
uuid: "82b026d06b454d65b27216e3002f29af"
format_stamp: "Formatted at 2025-10-28 08:53:42 on dist-test-slave-cmml"
I20251028 08:53:42.167235 5344 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20251028 08:53:42.170154 5361 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:42.171289 5344 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 08:53:42.171434 5344 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
uuid: "82b026d06b454d65b27216e3002f29af"
format_stamp: "Formatted at 2025-10-28 08:53:42 on dist-test-slave-cmml"
I20251028 08:53:42.171617 5344 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:42.197441 5344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:42.198184 5344 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:42.198428 5344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:42.205668 5343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } attempt: 4
W20251028 08:53:42.207571 5343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:53:42.217514 5344 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:37319
I20251028 08:53:42.217518 5412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.60:37319 every 8 connection(s)
I20251028 08:53:42.218827 5344 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
I20251028 08:53:42.222571 5413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:42.227108 5273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } attempt: 2
I20251028 08:53:42.227634 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 5344
I20251028 08:53:42.227860 4964 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal/instance
I20251028 08:53:42.228044 5413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
I20251028 08:53:42.240880 5413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } has no permanent_uuid. Determining permanent_uuid...
I20251028 08:53:42.245942 5273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Bootstrap starting.
I20251028 08:53:42.248684 5413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
I20251028 08:53:42.249382 5273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:42.250885 5273 log.cc:826] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:42.253482 5273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: No bootstrap required, opened a new log
I20251028 08:53:42.258658 5413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap starting.
I20251028 08:53:42.260409 5273 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.260731 5273 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:42.260892 5273 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9421be172b482f9c69d4dc5d2b19a8, State: Initialized, Role: FOLLOWER
I20251028 08:53:42.261750 5413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:42.261574 5273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.262972 5413 log.cc:826] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:42.263195 5421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.264110 5421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:42.264608 5273 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:53:42.268064 5413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: No bootstrap required, opened a new log
I20251028 08:53:42.274400 5413 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.274729 5413 raft_consensus.cc:385] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:42.274863 5413 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b026d06b454d65b27216e3002f29af, State: Initialized, Role: FOLLOWER
I20251028 08:53:42.275415 5413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.276335 5437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.276629 5437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:42.277377 5413 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:53:42.277321 5207 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: OK, instance_id { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" instance_seqno: 1761641621809391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251028 08:53:42.285147 5207 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:53:42.291896 5436 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251028 08:53:42.292062 5436 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251028 08:53:42.314097 5344 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: OK, instance_id { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" instance_seqno: 1761641621809391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251028 08:53:42.337038 5344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:53:42.340049 5454 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251028 08:53:42.340219 5454 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251028 08:53:42.391141 5227 tablet.cc:2378] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 08:53:42.450135 5367 tablet.cc:2378] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 08:53:42.500325 5437 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:42.500608 5437 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.502243 5437 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
I20251028 08:53:42.502969 5248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b026d06b454d65b27216e3002f29af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" is_pre_election: true
I20251028 08:53:42.503394 5248 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b026d06b454d65b27216e3002f29af in term 0.
I20251028 08:53:42.503943 5363 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 82b026d06b454d65b27216e3002f29af; no voters:
I20251028 08:53:42.504299 5437 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:42.504444 5437 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:42.504549 5437 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:42.506772 5343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } attempt: 5
I20251028 08:53:42.507649 5437 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.508318 5437 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 1 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
I20251028 08:53:42.509582 5248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b026d06b454d65b27216e3002f29af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
I20251028 08:53:42.509833 5248 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:42.513120 5248 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b026d06b454d65b27216e3002f29af in term 1.
I20251028 08:53:42.513736 5363 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 82b026d06b454d65b27216e3002f29af; no voters:
I20251028 08:53:42.514159 5437 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:42.519834 5343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap starting.
I20251028 08:53:42.523728 5343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:42.524973 5343 log.cc:826] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:42.530551 5343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: No bootstrap required, opened a new log
I20251028 08:53:42.536857 5437 raft_consensus.cc:697] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 1 LEADER]: Becoming Leader. State: Replica: 82b026d06b454d65b27216e3002f29af, State: Running, Role: LEADER
I20251028 08:53:42.537869 5437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.538118 5343 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.538389 5343 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:42.538512 5343 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Initialized, Role: FOLLOWER
I20251028 08:53:42.539212 5343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:42.548496 5456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b026d06b454d65b27216e3002f29af. Latest consensus state: current_term: 1 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.548795 5456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:42.552407 5460 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:53:42.556789 5460 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:53:42.562081 5458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.562461 5458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:42.563231 5318 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b026d06b454d65b27216e3002f29af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" is_pre_election: true
I20251028 08:53:42.563642 5318 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b026d06b454d65b27216e3002f29af in term 0.
I20251028 08:53:42.563944 5317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b026d06b454d65b27216e3002f29af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
I20251028 08:53:42.564175 5317 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:42.567253 5317 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b026d06b454d65b27216e3002f29af in term 1.
I20251028 08:53:42.567631 5343 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:53:42.577863 5248 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Refusing update from remote peer 82b026d06b454d65b27216e3002f29af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 08:53:42.579095 5456 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:42.579540 5317 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 1 FOLLOWER]: Refusing update from remote peer 82b026d06b454d65b27216e3002f29af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 08:53:42.580462 5456 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:42.599581 5421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b026d06b454d65b27216e3002f29af. Latest consensus state: current_term: 1 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.599856 5421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:42.606439 5458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b026d06b454d65b27216e3002f29af. Latest consensus state: current_term: 1 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.606681 5458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:42.608444 5461 mvcc.cc:204] Tried to move back new op lower bound from 7215684086067810304 to 7215684085927555072. Current Snapshot: MvccSnapshot[applied={T|T < 7215684086067810304}]
I20251028 08:53:42.614029 5456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.614241 5456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:42.614758 5458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.614944 5458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:42.615651 5466 mvcc.cc:204] Tried to move back new op lower bound from 7215684086067810304 to 7215684085927555072. Current Snapshot: MvccSnapshot[applied={T|T < 7215684086067810304}]
I20251028 08:53:42.616726 5456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.616907 5456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:42.618466 5463 mvcc.cc:204] Tried to move back new op lower bound from 7215684086067810304 to 7215684085927555072. Current Snapshot: MvccSnapshot[applied={T|T < 7215684086067810304}]
I20251028 08:53:42.622082 5421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:42.622241 5460 catalog_manager.cc:1357] Generated new cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:42.622254 5421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:42.622327 5460 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:53:42.661139 5460 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 08:53:42.662629 5460 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:53:42.685370 5460 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Generated new TSK 0
I20251028 08:53:42.686347 5460 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:53:42.687520 5460 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:53:42.688587 5460 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:53:42.713315 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.1:0
--local_ip_for_outbound_sockets=127.4.217.1
--webserver_interface=127.4.217.1
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251028 08:53:42.734323 5484 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:42.734431 5484 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: loading cluster ID for follower catalog manager: success
I20251028 08:53:42.742209 5484 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: acquiring CA information for follower catalog manager: success
I20251028 08:53:42.744681 5484 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251028 08:53:43.079119 5486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:43.079394 5486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:43.079567 5486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:43.087237 5486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:43.087395 5486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.1
I20251028 08:53:43.095682 5486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.217.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.1
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:43.097040 5486 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:43.098647 5486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:43.107519 5492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:43.108116 5491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:43.109061 5494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:43.109414 5486 server_base.cc:1047] running on GCE node
I20251028 08:53:43.110144 5486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:43.111166 5486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:43.112406 5486 hybrid_clock.cc:648] HybridClock initialized: now 1761641623112358 us; error 37 us; skew 500 ppm
I20251028 08:53:43.112689 5486 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:43.114938 5486 webserver.cc:492] Webserver started at http://127.4.217.1:42065/ using document root <none> and password file <none>
I20251028 08:53:43.115761 5486 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:43.115854 5486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:43.116217 5486 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 08:53:43.118865 5486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data/instance:
uuid: "418c8c132d8141ae96893d82e0217aed"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.119779 5486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/instance:
uuid: "418c8c132d8141ae96893d82e0217aed"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.125440 5486 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251028 08:53:43.129034 5500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:43.130460 5486 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251028 08:53:43.130630 5486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal
uuid: "418c8c132d8141ae96893d82e0217aed"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.130853 5486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:43.149420 5486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:43.150300 5486 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:43.150555 5486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:43.151535 5486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251028 08:53:43.153394 5486 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 08:53:43.153496 5486 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:43.153585 5486 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 08:53:43.153676 5486 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251028 08:53:43.184999 5608 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82b026d06b454d65b27216e3002f29af)
I20251028 08:53:43.185209 5608 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:43.186458 5608 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:53:43.186899 5317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" is_pre_election: true
I20251028 08:53:43.187173 5388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af" is_pre_election: true
I20251028 08:53:43.187956 5486 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.1:42085
I20251028 08:53:43.188019 5613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.1:42085 every 8 connection(s)
I20251028 08:53:43.188025 5222 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:43.188344 5608 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251028 08:53:43.189383 5486 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/data/info.pb
I20251028 08:53:43.197731 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 5486
I20251028 08:53:43.197980 4964 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/instance
I20251028 08:53:43.203650 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.2:0
--local_ip_for_outbound_sockets=127.4.217.2
--webserver_interface=127.4.217.2
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251028 08:53:43.227218 5614 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:43.227545 5614 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:43.228399 5614 heartbeater.cc:507] Master 127.4.217.60:37319 requested a full tablet report, sending...
I20251028 08:53:43.228466 5616 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:53:43.228606 5616 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:43.229142 5616 heartbeater.cc:507] Master 127.4.217.61:33257 requested a full tablet report, sending...
I20251028 08:53:43.230360 5377 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:43.230975 5308 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:43.232363 5377 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.4.217.1:51785
I20251028 08:53:43.235724 5615 heartbeater.cc:344] Connected to a master server at 127.4.217.62:41357
I20251028 08:53:43.235845 5615 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:43.236160 5615 heartbeater.cc:507] Master 127.4.217.62:41357 requested a full tablet report, sending...
I20251028 08:53:43.237550 5238 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:43.294121 5436 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:43.294214 5436 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: loading cluster ID for follower catalog manager: success
I20251028 08:53:43.297546 5436 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: acquiring CA information for follower catalog manager: success
I20251028 08:53:43.299525 5436 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251028 08:53:43.515014 5624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:43.515257 5624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:43.515334 5624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:43.522522 5624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:43.522650 5624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.2
I20251028 08:53:43.530748 5624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.217.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.2
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:43.531985 5624 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:43.533424 5624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:43.541946 5629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:43.542181 5630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:43.542323 5632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:43.543262 5624 server_base.cc:1047] running on GCE node
I20251028 08:53:43.543737 5624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:43.544453 5624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:43.545678 5624 hybrid_clock.cc:648] HybridClock initialized: now 1761641623545621 us; error 28 us; skew 500 ppm
I20251028 08:53:43.545926 5624 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:43.547663 5624 webserver.cc:492] Webserver started at http://127.4.217.2:36049/ using document root <none> and password file <none>
I20251028 08:53:43.548247 5624 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:43.548339 5624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:43.548645 5624 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 08:53:43.550822 5624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data/instance:
uuid: "52a17de1c37d4a339d1fcb35b550aad7"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.551506 5624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/instance:
uuid: "52a17de1c37d4a339d1fcb35b550aad7"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.555882 5624 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251028 08:53:43.558867 5638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:43.559973 5624 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 08:53:43.560120 5624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal
uuid: "52a17de1c37d4a339d1fcb35b550aad7"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.560317 5624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:43.574478 5624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:43.575104 5624 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:43.575318 5624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:43.576115 5624 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251028 08:53:43.577507 5624 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 08:53:43.577636 5624 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:43.577728 5624 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 08:53:43.577790 5624 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:43.607527 5624 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.2:35819
I20251028 08:53:43.607583 5750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.2:35819 every 8 connection(s)
I20251028 08:53:43.608718 5624 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/data/info.pb
I20251028 08:53:43.611958 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 5624
I20251028 08:53:43.612169 4964 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/instance
I20251028 08:53:43.617470 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.217.3:0
--local_ip_for_outbound_sockets=127.4.217.3
--webserver_interface=127.4.217.3
--webserver_port=0
--tserver_master_addrs=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251028 08:53:43.631357 5753 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:53:43.631498 5752 heartbeater.cc:344] Connected to a master server at 127.4.217.62:41357
I20251028 08:53:43.631877 5752 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:43.632823 5752 heartbeater.cc:507] Master 127.4.217.62:41357 requested a full tablet report, sending...
I20251028 08:53:43.634318 5751 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:43.634336 5753 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:43.634447 5751 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:43.634578 5238 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:43.634994 5751 heartbeater.cc:507] Master 127.4.217.60:37319 requested a full tablet report, sending...
I20251028 08:53:43.635140 5753 heartbeater.cc:507] Master 127.4.217.61:33257 requested a full tablet report, sending...
I20251028 08:53:43.635802 5377 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:43.636096 5308 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:43.636649 5377 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.4.217.2:50605
I20251028 08:53:43.735141 5614 heartbeater.cc:499] Master 127.4.217.60:37319 was elected leader, sending a full tablet report...
W20251028 08:53:43.912408 5758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:43.912624 5758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:43.912744 5758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:43.920047 5758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:43.920229 5758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.217.3
I20251028 08:53:43.928670 5758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.217.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.217.3
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:43.929968 5758 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:43.931600 5758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:43.940001 5763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:43.940028 5764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:43.940642 5766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:43.941473 5758 server_base.cc:1047] running on GCE node
I20251028 08:53:43.942106 5758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:43.942898 5758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:43.944077 5758 hybrid_clock.cc:648] HybridClock initialized: now 1761641623944036 us; error 28 us; skew 500 ppm
I20251028 08:53:43.944329 5758 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:43.946126 5758 webserver.cc:492] Webserver started at http://127.4.217.3:36489/ using document root <none> and password file <none>
I20251028 08:53:43.946731 5758 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:43.946815 5758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:43.947139 5758 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 08:53:43.949086 5758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data/instance:
uuid: "bbab3d08643e49e786c2174faaa741d9"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.949808 5758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/instance:
uuid: "bbab3d08643e49e786c2174faaa741d9"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.954382 5758 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251028 08:53:43.957235 5772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:43.958331 5758 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251028 08:53:43.958470 5758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal
uuid: "bbab3d08643e49e786c2174faaa741d9"
format_stamp: "Formatted at 2025-10-28 08:53:43 on dist-test-slave-cmml"
I20251028 08:53:43.958710 5758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:43.975515 5758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:43.976191 5758 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:43.976423 5758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:43.977191 5758 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251028 08:53:43.978688 5758 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 08:53:43.978785 5758 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:43.978895 5758 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 08:53:43.978964 5758 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:44.009949 5758 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.3:33455
I20251028 08:53:44.010006 5884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.3:33455 every 8 connection(s)
I20251028 08:53:44.011165 5758 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/data/info.pb
I20251028 08:53:44.012902 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 5758
I20251028 08:53:44.013075 4964 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/instance
I20251028 08:53:44.034087 5885 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:44.034111 5886 heartbeater.cc:344] Connected to a master server at 127.4.217.62:41357
I20251028 08:53:44.034163 5887 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:53:44.034523 5885 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:44.034521 5886 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:44.034525 5887 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:44.035508 5886 heartbeater.cc:507] Master 127.4.217.62:41357 requested a full tablet report, sending...
I20251028 08:53:44.035660 5887 heartbeater.cc:507] Master 127.4.217.61:33257 requested a full tablet report, sending...
I20251028 08:53:44.036477 5885 heartbeater.cc:507] Master 127.4.217.60:37319 requested a full tablet report, sending...
I20251028 08:53:44.037397 5308 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:44.037951 5238 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:44.037979 5378 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:44.038712 5378 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.4.217.3:42113
I20251028 08:53:44.041862 5892 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82b026d06b454d65b27216e3002f29af)
I20251028 08:53:44.042011 5892 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:44.042984 5892 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:53:44.043323 5388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af" is_pre_election: true
I20251028 08:53:44.043339 5248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" is_pre_election: true
I20251028 08:53:44.043846 5292 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 82b026d06b454d65b27216e3002f29af
I20251028 08:53:44.044147 5892 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251028 08:53:44.051781 4964 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 08:53:44.087049 5377 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.087077 5378 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_7991714d6af648189d6314e21f58740e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.088554 5376 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_64f349518d5d449ea6a147f1795a24fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.089226 5375 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_78fd665055cb412390a37c5a185dec85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.089944 5374 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_976045e19bfe41c3b314c8cc12e020d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.090643 5373 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_dd23f4b28a514540926eb05e3f12c357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.091249 5372 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_f617dc4ea4f444fa8528dda685e72fa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.091885 5371 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_a90e8e5ec3364d6b80abc305644d234c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.092482 5370 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_d77cbfbaef6940f4bd623eaab2bb30aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251028 08:53:44.093668 5377 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.096752 5376 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_64f349518d5d449ea6a147f1795a24fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.097687 5375 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_78fd665055cb412390a37c5a185dec85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.097995 5371 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a90e8e5ec3364d6b80abc305644d234c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.098570 5378 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7991714d6af648189d6314e21f58740e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.099170 5373 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dd23f4b28a514540926eb05e3f12c357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.100402 5372 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f617dc4ea4f444fa8528dda685e72fa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.101612 5374 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_976045e19bfe41c3b314c8cc12e020d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 08:53:44.103371 5369 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40892:
name: "default.Table_6d9d93bde7fa47158f8e7b574783370e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.111186 4964 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1hpXWl/build/asan/bin/kudu pid 5344
I20251028 08:53:44.112354 5408 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 08:53:44.138355 5751 heartbeater.cc:499] Master 127.4.217.60:37319 was elected leader, sending a full tablet report...
I20251028 08:53:44.177520 5933 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82b026d06b454d65b27216e3002f29af)
I20251028 08:53:44.177762 5933 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:44.178794 5933 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:53:44.178817 5317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" is_pre_election: true
I20251028 08:53:44.179039 5317 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b9421be172b482f9c69d4dc5d2b19a8 in term 1.
I20251028 08:53:44.179528 5224 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters:
I20251028 08:53:44.187194 5933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251028 08:53:44.187299 5933 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82b026d06b454d65b27216e3002f29af)
I20251028 08:53:44.187384 5933 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 1 FOLLOWER]: Advancing to term 2
I20251028 08:53:44.190193 5933 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:44.191113 5317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
I20251028 08:53:44.191299 5317 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 1 FOLLOWER]: Advancing to term 2
I20251028 08:53:44.193986 5317 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b9421be172b482f9c69d4dc5d2b19a8 in term 2.
I20251028 08:53:44.194458 5224 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters:
I20251028 08:53:44.200733 5933 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 election: Requested vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:53:44.201045 5933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 FOLLOWER]: Leader election won for term 2
I20251028 08:53:44.201385 5933 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 LEADER]: Becoming Leader. State: Replica: 0b9421be172b482f9c69d4dc5d2b19a8, State: Running, Role: LEADER
I20251028 08:53:44.201992 5933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:44.212925 5933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b9421be172b482f9c69d4dc5d2b19a8. Latest consensus state: current_term: 2 leader_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:44.213555 5933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:44.215355 5936 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:53:44.217464 5936 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:53:44.224344 5936 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:44.224522 5936 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:53:44.225801 5936 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:53:44.226848 5936 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Loaded TSK: 0
I20251028 08:53:44.227955 5936 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:53:44.228864 5936 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:53:44.229878 5936 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:53:44.256810 5248 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 LEADER]: Rejecting Update request from peer 82b026d06b454d65b27216e3002f29af for earlier term 1. Current term is 2. Ops: []
I20251028 08:53:44.257800 4964 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1hpXWl/build/asan/bin/kudu with pid 5344
W20251028 08:53:44.283162 5614 heartbeater.cc:646] Failed to heartbeat to 127.4.217.60:37319 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.217.60:37319 (error 108)
W20251028 08:53:44.283456 5893 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.60:37319)
W20251028 08:53:44.283792 5223 connection.cc:537] server connection from 127.0.0.1:40292 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:44.286103 5751 heartbeater.cc:646] Failed to heartbeat to 127.4.217.60:37319 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.217.60:37319 (error 108)
W20251028 08:53:44.286302 5222 connection.cc:537] client connection to 127.4.217.60:37319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 08:53:44.291818 5235 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_f617dc4ea4f444fa8528dda685e72fa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.292277 5238 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_d77cbfbaef6940f4bd623eaab2bb30aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.292897 5233 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_7991714d6af648189d6314e21f58740e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.293002 5236 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_a90e8e5ec3364d6b80abc305644d234c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.293637 5231 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_78fd665055cb412390a37c5a185dec85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.293932 5232 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_64f349518d5d449ea6a147f1795a24fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.294305 5229 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_dd23f4b28a514540926eb05e3f12c357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.294711 5230 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_976045e19bfe41c3b314c8cc12e020d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.295497 5234 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:44.296824 5237 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_6d9d93bde7fa47158f8e7b574783370e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251028 08:53:44.304026 5222 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:53:44.304383 5222 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:53:44.307765 5235 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f617dc4ea4f444fa8528dda685e72fa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.307785 5233 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7991714d6af648189d6314e21f58740e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.307780 5229 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dd23f4b28a514540926eb05e3f12c357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.307981 5238 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d77cbfbaef6940f4bd623eaab2bb30aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.308306 5230 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_976045e19bfe41c3b314c8cc12e020d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.308732 5232 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_64f349518d5d449ea6a147f1795a24fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.310480 5236 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a90e8e5ec3364d6b80abc305644d234c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:44.311582 5231 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_78fd665055cb412390a37c5a185dec85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 08:53:44.541198 5885 heartbeater.cc:499] Master 127.4.217.60:37319 was elected leader, sending a full tablet report...
W20251028 08:53:44.542740 5885 heartbeater.cc:646] Failed to heartbeat to 127.4.217.60:37319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:53:44.598505 5230 catalog_manager.cc:2174] created HMS catalog entry for table [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:53:44.602857 5317 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 2 FOLLOWER]: Refusing update from remote peer 0b9421be172b482f9c69d4dc5d2b19a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20251028 08:53:44.603051 5222 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 -> Peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Couldn't send request to peer 82b026d06b454d65b27216e3002f29af. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251028 08:53:44.603688 5933 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251028 08:53:44.620520 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b9421be172b482f9c69d4dc5d2b19a8. Latest consensus state: current_term: 2 leader_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:44.620745 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:44.630172 5933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:44.630411 5933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:44.637748 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:44.637943 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:44.639631 5230 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40314:
table { table_name: "default.Table_976045e19bfe41c3b314c8cc12e020d1" } new_table_name: "default.Table_dfd48b4fd93d407bad513cecd15ecf68" modify_external_catalogs: true
I20251028 08:53:44.667546 5686 tablet_service.cc:1505] Processing CreateTablet for tablet 81d606fd4f6c4576bf885b3169b46901 (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.668915 5686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d606fd4f6c4576bf885b3169b46901. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.674012 5548 tablet_service.cc:1505] Processing CreateTablet for tablet 81d606fd4f6c4576bf885b3169b46901 (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.675290 5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d606fd4f6c4576bf885b3169b46901. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.684474 5820 tablet_service.cc:1505] Processing CreateTablet for tablet 81d606fd4f6c4576bf885b3169b46901 (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.685731 5820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d606fd4f6c4576bf885b3169b46901. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.687240 5238 catalog_manager.cc:2174] created HMS catalog entry for table [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:53:44.701191 5954 tablet_bootstrap.cc:492] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:44.706861 5686 tablet_service.cc:1505] Processing CreateTablet for tablet 4289997e6a7f459187743edfe904bd66 (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.706923 5547 tablet_service.cc:1505] Processing CreateTablet for tablet 4289997e6a7f459187743edfe904bd66 (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.707223 5686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4289997e6a7f459187743edfe904bd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.707312 5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4289997e6a7f459187743edfe904bd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.710903 5238 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40314:
table { table_name: "default.Table_d77cbfbaef6940f4bd623eaab2bb30aa" } new_table_name: "default.Table_a32163429a914f28ad8f6bac7f7c11c0" modify_external_catalogs: true
I20251028 08:53:44.711011 5819 tablet_service.cc:1505] Processing CreateTablet for tablet 4289997e6a7f459187743edfe904bd66 (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.711407 5819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4289997e6a7f459187743edfe904bd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.721773 5953 tablet_bootstrap.cc:492] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:44.725687 5953 tablet_bootstrap.cc:654] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.726903 5953 log.cc:826] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:44.732931 5955 tablet_bootstrap.cc:492] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:44.737229 5954 tablet_bootstrap.cc:654] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.738565 5954 log.cc:826] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:44.739017 5955 tablet_bootstrap.cc:654] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.739353 5953 tablet_bootstrap.cc:492] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:44.739627 5953 ts_tablet_manager.cc:1403] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.004s
I20251028 08:53:44.740859 5955 log.cc:826] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:44.741765 5615 heartbeater.cc:499] Master 127.4.217.62:41357 was elected leader, sending a full tablet report...
I20251028 08:53:44.753903 5232 catalog_manager.cc:2174] created HMS catalog entry for table [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:53:44.756877 5953 raft_consensus.cc:359] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.757292 5953 raft_consensus.cc:385] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.757391 5953 raft_consensus.cc:740] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.758941 5953 consensus_queue.cc:260] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.760504 5954 tablet_bootstrap.cc:492] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:44.760784 5954 ts_tablet_manager.cc:1403] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.015s
I20251028 08:53:44.767978 5954 raft_consensus.cc:359] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.768337 5954 raft_consensus.cc:385] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.768427 5954 raft_consensus.cc:740] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.770391 5232 master_service.cc:954] ReplaceTablet: received request to replace tablet 4289997e6a7f459187743edfe904bd66 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:44.771281 5955 tablet_bootstrap.cc:492] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:44.771605 5955 ts_tablet_manager.cc:1403] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.006s
I20251028 08:53:44.773337 5548 tablet_service.cc:1505] Processing CreateTablet for tablet 7ff55e6d288f44428891d6fc7e93e2d2 (DEFAULT_TABLE table=default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.773921 5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff55e6d288f44428891d6fc7e93e2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.775331 5686 tablet_service.cc:1505] Processing CreateTablet for tablet 7ff55e6d288f44428891d6fc7e93e2d2 (DEFAULT_TABLE table=default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.775743 5686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff55e6d288f44428891d6fc7e93e2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.778939 5232 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Sending DeleteTablet for 3 replicas of tablet 4289997e6a7f459187743edfe904bd66
I20251028 08:53:44.779709 5232 catalog_manager.cc:6719] ReplaceTablet: tablet 4289997e6a7f459187743edfe904bd66 deleted and replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7
I20251028 08:53:44.780009 5232 master_service.cc:954] ReplaceTablet: received request to replace tablet 4289997e6a7f459187743edfe904bd66 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:44.780216 5547 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:54274
I20251028 08:53:44.780393 5820 tablet_service.cc:1505] Processing CreateTablet for tablet 7ff55e6d288f44428891d6fc7e93e2d2 (DEFAULT_TABLE table=default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.780609 5232 master_service.cc:954] ReplaceTablet: received request to replace tablet 81d606fd4f6c4576bf885b3169b46901 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:44.780772 5820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff55e6d288f44428891d6fc7e93e2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.781562 5819 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:44.787717 5232 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Sending DeleteTablet for 3 replicas of tablet 81d606fd4f6c4576bf885b3169b46901
I20251028 08:53:44.793116 5955 raft_consensus.cc:359] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.793481 5955 raft_consensus.cc:385] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.793910 5685 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:38788
I20251028 08:53:44.796684 5232 catalog_manager.cc:6719] ReplaceTablet: tablet 81d606fd4f6c4576bf885b3169b46901 deleted and replaced by tablet 93164793833348ef97b3d6f7fe53ba8d
I20251028 08:53:44.796677 5684 tablet_service.cc:1505] Processing CreateTablet for tablet 2c248e923a4f4085973a41a96fd1e0b7 (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.797148 5684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c248e923a4f4085973a41a96fd1e0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.797130 5817 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d606fd4f6c4576bf885b3169b46901 with delete_type TABLET_DATA_DELETED (replaced by tablet 93164793833348ef97b3d6f7fe53ba8d) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:44.798749 5232 master_service.cc:954] ReplaceTablet: received request to replace tablet 81d606fd4f6c4576bf885b3169b46901 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:44.799352 5232 master_service.cc:954] ReplaceTablet: received request to replace tablet 4289997e6a7f459187743edfe904bd66 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:44.801946 5818 tablet_service.cc:1505] Processing CreateTablet for tablet 2c248e923a4f4085973a41a96fd1e0b7 (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.802431 5818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c248e923a4f4085973a41a96fd1e0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.803247 5954 consensus_queue.cc:260] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.806953 5955 raft_consensus.cc:740] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.807741 5232 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40314:
table { table_name: "default.Table_64f349518d5d449ea6a147f1795a24fd" } new_table_name: "default.Table_018ba9cb543a4fd0a8e19856261027a1" modify_external_catalogs: true
I20251028 08:53:44.807832 5683 tablet_service.cc:1505] Processing CreateTablet for tablet 93164793833348ef97b3d6f7fe53ba8d (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.808290 5683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93164793833348ef97b3d6f7fe53ba8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251028 08:53:44.810683 5222 catalog_manager.cc:4965] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): delete failed for tablet 4289997e6a7f459187743edfe904bd66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4289997e6a7f459187743edfe904bd66 already in progress: creating tablet
I20251028 08:53:44.816865 5686 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d606fd4f6c4576bf885b3169b46901 with delete_type TABLET_DATA_DELETED (replaced by tablet 93164793833348ef97b3d6f7fe53ba8d) from {username='slave'} at 127.0.0.1:38788
W20251028 08:53:44.817447 5222 catalog_manager.cc:4965] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): delete failed for tablet 81d606fd4f6c4576bf885b3169b46901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81d606fd4f6c4576bf885b3169b46901 already in progress: creating tablet
I20251028 08:53:44.817977 5820 tablet_service.cc:1505] Processing CreateTablet for tablet 93164793833348ef97b3d6f7fe53ba8d (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.818313 5820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93164793833348ef97b3d6f7fe53ba8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.825201 5955 consensus_queue.cc:260] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.825443 5545 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d606fd4f6c4576bf885b3169b46901 with delete_type TABLET_DATA_DELETED (replaced by tablet 93164793833348ef97b3d6f7fe53ba8d) from {username='slave'} at 127.0.0.1:54274
I20251028 08:53:44.826251 5544 tablet_service.cc:1505] Processing CreateTablet for tablet 93164793833348ef97b3d6f7fe53ba8d (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
W20251028 08:53:44.826726 5223 catalog_manager.cc:4965] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): delete failed for tablet 4289997e6a7f459187743edfe904bd66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4289997e6a7f459187743edfe904bd66 already in progress: creating tablet
I20251028 08:53:44.826771 5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93164793833348ef97b3d6f7fe53ba8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251028 08:53:44.827147 5223 catalog_manager.cc:4965] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): delete failed for tablet 81d606fd4f6c4576bf885b3169b46901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81d606fd4f6c4576bf885b3169b46901 already in progress: creating tablet
I20251028 08:53:44.829614 5954 ts_tablet_manager.cc:1434] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.069s user 0.011s sys 0.000s
I20251028 08:53:44.829733 5752 heartbeater.cc:499] Master 127.4.217.62:41357 was elected leader, sending a full tablet report...
I20251028 08:53:44.830016 5954 tablet_bootstrap.cc:492] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:44.831790 5954 tablet_bootstrap.cc:654] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.834017 5546 tablet_service.cc:1505] Processing CreateTablet for tablet 2c248e923a4f4085973a41a96fd1e0b7 (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.837631 5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c248e923a4f4085973a41a96fd1e0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.844965 5954 tablet_bootstrap.cc:492] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:44.845117 5954 ts_tablet_manager.cc:1403] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.002s
I20251028 08:53:44.846104 5954 raft_consensus.cc:359] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.846285 5954 raft_consensus.cc:385] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.846366 5954 raft_consensus.cc:740] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.846617 5954 consensus_queue.cc:260] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.847998 5954 ts_tablet_manager.cc:1434] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251028 08:53:44.849020 5954 tablet_bootstrap.cc:492] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:44.850734 5954 tablet_bootstrap.cc:654] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.854650 5953 ts_tablet_manager.cc:1434] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.115s user 0.015s sys 0.003s
I20251028 08:53:44.861338 5953 tablet_bootstrap.cc:492] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
W20251028 08:53:44.861614 5754 tablet.cc:2378] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 08:53:44.867074 5960 tablet_replica.cc:333] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
W20251028 08:53:44.867144 5222 catalog_manager.cc:4965] TS 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): delete failed for tablet 4289997e6a7f459187743edfe904bd66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4289997e6a7f459187743edfe904bd66 already in progress: creating tablet
I20251028 08:53:44.867606 5960 raft_consensus.cc:2243] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:44.868067 5960 raft_consensus.cc:2272] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:44.871196 5953 tablet_bootstrap.cc:654] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.871536 5960 ts_tablet_manager.cc:1916] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:44.885519 5953 tablet_bootstrap.cc:492] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:44.885730 5960 ts_tablet_manager.cc:1929] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:44.885851 5960 log.cc:1199] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/81d606fd4f6c4576bf885b3169b46901
I20251028 08:53:44.885731 5953 ts_tablet_manager.cc:1403] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20251028 08:53:44.886265 5960 ts_tablet_manager.cc:1950] T 81d606fd4f6c4576bf885b3169b46901 P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:53:44.887413 5953 raft_consensus.cc:359] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.887956 5222 catalog_manager.cc:4985] TS 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): tablet 81d606fd4f6c4576bf885b3169b46901 (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]) successfully deleted
I20251028 08:53:44.888684 5886 heartbeater.cc:499] Master 127.4.217.62:41357 was elected leader, sending a full tablet report...
I20251028 08:53:44.889410 5955 ts_tablet_manager.cc:1434] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.118s user 0.015s sys 0.036s
I20251028 08:53:44.889700 5953 raft_consensus.cc:385] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.889827 5953 raft_consensus.cc:740] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.890084 5953 consensus_queue.cc:260] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.890581 5953 ts_tablet_manager.cc:1434] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251028 08:53:44.890884 5953 tablet_bootstrap.cc:492] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:44.895596 5955 tablet_bootstrap.cc:492] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:44.897130 5953 tablet_bootstrap.cc:654] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.899063 5954 tablet_bootstrap.cc:492] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:44.899222 5954 ts_tablet_manager.cc:1403] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.016s
I20251028 08:53:44.899953 5954 raft_consensus.cc:359] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.900152 5954 raft_consensus.cc:385] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.900260 5954 raft_consensus.cc:740] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.900318 5955 tablet_bootstrap.cc:654] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.900538 5954 consensus_queue.cc:260] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.901010 5954 ts_tablet_manager.cc:1434] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251028 08:53:44.904654 5954 tablet_bootstrap.cc:492] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:44.907321 5954 tablet_bootstrap.cc:654] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.908845 5953 tablet_bootstrap.cc:492] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:44.909003 5953 ts_tablet_manager.cc:1403] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20251028 08:53:44.909703 5953 raft_consensus.cc:359] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.909890 5953 raft_consensus.cc:385] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.909982 5953 raft_consensus.cc:740] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.910243 5953 consensus_queue.cc:260] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.912276 5953 ts_tablet_manager.cc:1434] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251028 08:53:44.914281 5953 tablet_bootstrap.cc:492] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:44.915094 5229 catalog_manager.cc:2174] created HMS catalog entry for table [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:53:44.915670 5953 tablet_bootstrap.cc:654] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.917052 5954 tablet_bootstrap.cc:492] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:44.917208 5954 ts_tablet_manager.cc:1403] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251028 08:53:44.918105 5954 raft_consensus.cc:359] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.918344 5954 raft_consensus.cc:385] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.918428 5954 raft_consensus.cc:740] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.918653 5954 consensus_queue.cc:260] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.919189 5954 ts_tablet_manager.cc:1434] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 08:53:44.919566 5954 tablet_bootstrap.cc:492] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:44.921782 5954 tablet_bootstrap.cc:654] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.923902 5955 tablet_bootstrap.cc:492] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:44.924069 5955 ts_tablet_manager.cc:1403] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.012s
I20251028 08:53:44.925181 5955 raft_consensus.cc:359] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.925395 5955 raft_consensus.cc:385] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:44.925490 5955 raft_consensus.cc:740] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:44.925801 5955 consensus_queue.cc:260] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.929262 5955 ts_tablet_manager.cc:1434] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251028 08:53:44.932327 5955 tablet_bootstrap.cc:492] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:44.935987 5955 tablet_bootstrap.cc:654] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:44.938277 5546 tablet_service.cc:1505] Processing CreateTablet for tablet 633ad9f0d80c485fbe8d7bbff88f3d58 (DEFAULT_TABLE table=default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.938730 5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 633ad9f0d80c485fbe8d7bbff88f3d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.941962 5685 tablet_service.cc:1505] Processing CreateTablet for tablet 633ad9f0d80c485fbe8d7bbff88f3d58 (DEFAULT_TABLE table=default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.942559 5685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 633ad9f0d80c485fbe8d7bbff88f3d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.947001 5819 tablet_service.cc:1505] Processing CreateTablet for tablet 633ad9f0d80c485fbe8d7bbff88f3d58 (DEFAULT_TABLE table=default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:44.947399 5819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 633ad9f0d80c485fbe8d7bbff88f3d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:44.951225 5964 raft_consensus.cc:493] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:44.951443 5964 raft_consensus.cc:515] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.953266 5964 leader_election.cc:290] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:44.957080 5684 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:38788
I20251028 08:53:44.957423 5963 tablet_replica.cc:333] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:53:44.957958 5963 raft_consensus.cc:2243] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:44.958303 5963 raft_consensus.cc:2272] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:44.961154 5963 ts_tablet_manager.cc:1916] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251028 08:53:44.967903 5617 tablet.cc:2378] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 08:53:44.969667 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ff55e6d288f44428891d6fc7e93e2d2" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
W20251028 08:53:44.970664 5640 leader_election.cc:343] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Illegal state: must be running to vote when last-logged opid is not known
I20251028 08:53:44.974968 5684 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d606fd4f6c4576bf885b3169b46901 with delete_type TABLET_DATA_DELETED (replaced by tablet 93164793833348ef97b3d6f7fe53ba8d) from {username='slave'} at 127.0.0.1:38788
I20251028 08:53:44.979887 5964 raft_consensus.cc:493] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:44.980106 5964 raft_consensus.cc:515] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.980873 5964 leader_election.cc:290] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:44.981900 5966 raft_consensus.cc:493] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:44.982089 5966 raft_consensus.cc:515] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:44.982820 5548 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d606fd4f6c4576bf885b3169b46901 with delete_type TABLET_DATA_DELETED (replaced by tablet 93164793833348ef97b3d6f7fe53ba8d) from {username='slave'} at 127.0.0.1:54274
W20251028 08:53:44.983336 5222 catalog_manager.cc:4952] TS 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): delete failed for tablet 81d606fd4f6c4576bf885b3169b46901 because the tablet was not found. No further retry: Not found: Tablet not found: 81d606fd4f6c4576bf885b3169b46901
I20251028 08:53:44.983706 5966 leader_election.cc:290] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:44.985008 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c248e923a4f4085973a41a96fd1e0b7" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
W20251028 08:53:44.986411 5640 leader_election.cc:343] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Illegal state: must be running to vote when last-logged opid is not known
I20251028 08:53:44.997515 5819 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:44.997810 5961 tablet_replica.cc:333] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:53:44.997946 5819 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d606fd4f6c4576bf885b3169b46901 with delete_type TABLET_DATA_DELETED (replaced by tablet 93164793833348ef97b3d6f7fe53ba8d) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:44.998246 5961 raft_consensus.cc:2243] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:44.998680 5963 ts_tablet_manager.cc:1929] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:44.998679 5961 raft_consensus.cc:2272] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:44.998826 5963 log.cc:1199] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/4289997e6a7f459187743edfe904bd66
I20251028 08:53:44.999215 5963 ts_tablet_manager.cc:1950] T 4289997e6a7f459187743edfe904bd66 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
I20251028 08:53:45.000795 5222 catalog_manager.cc:4985] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): tablet 4289997e6a7f459187743edfe904bd66 (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]) successfully deleted
I20251028 08:53:45.000998 5961 ts_tablet_manager.cc:1916] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251028 08:53:45.007764 5773 leader_election.cc:343] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 81d606fd4f6c4576bf885b3169b46901
I20251028 08:53:45.008070 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--webserver_interface=127.4.217.60
--webserver_port=41817
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251028 08:53:45.012109 5685 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:38788
W20251028 08:53:45.013860 5888 tablet.cc:2378] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 08:53:45.016908 5546 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:54274
I20251028 08:53:45.017202 5960 tablet_replica.cc:333] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
I20251028 08:53:45.017509 5960 raft_consensus.cc:2243] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.017729 5960 raft_consensus.cc:2272] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.018438 5960 ts_tablet_manager.cc:1916] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.020396 5961 ts_tablet_manager.cc:1929] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.021253 5961 log.cc:1199] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/4289997e6a7f459187743edfe904bd66
I20251028 08:53:45.021718 5961 ts_tablet_manager.cc:1950] T 4289997e6a7f459187743edfe904bd66 P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
I20251028 08:53:45.022974 5961 tablet_replica.cc:333] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:53:45.023059 5223 catalog_manager.cc:4985] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): tablet 4289997e6a7f459187743edfe904bd66 (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]) successfully deleted
I20251028 08:53:45.025746 5961 raft_consensus.cc:2243] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.025939 5961 raft_consensus.cc:2272] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.026566 5961 ts_tablet_manager.cc:1916] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.026661 5960 ts_tablet_manager.cc:1929] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.026820 5960 log.cc:1199] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/4289997e6a7f459187743edfe904bd66
I20251028 08:53:45.027195 5960 ts_tablet_manager.cc:1950] T 4289997e6a7f459187743edfe904bd66 P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:53:45.028210 5222 catalog_manager.cc:4985] TS 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): tablet 4289997e6a7f459187743edfe904bd66 (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]) successfully deleted
I20251028 08:53:45.030838 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ff55e6d288f44428891d6fc7e93e2d2" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.031227 5568 raft_consensus.cc:2468] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 0.
I20251028 08:53:45.031908 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c248e923a4f4085973a41a96fd1e0b7" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.032519 5639 leader_election.cc:304] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7; no voters: bbab3d08643e49e786c2174faaa741d9
I20251028 08:53:45.032929 5964 raft_consensus.cc:2804] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:45.033037 5964 raft_consensus.cc:493] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:45.033121 5964 raft_consensus.cc:3060] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.035774 5964 raft_consensus.cc:515] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.037021 5964 leader_election.cc:290] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Requested vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.037011 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ff55e6d288f44428891d6fc7e93e2d2" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9"
W20251028 08:53:45.037429 5640 leader_election.cc:343] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Illegal state: must be running to vote when last-logged opid is not known
W20251028 08:53:45.037737 5639 leader_election.cc:343] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Illegal state: must be running to vote when last-logged opid is not known
I20251028 08:53:45.037955 5639 leader_election.cc:304] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7; no voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9
I20251028 08:53:45.038267 5964 raft_consensus.cc:2749] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:45.039063 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ff55e6d288f44428891d6fc7e93e2d2" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed"
I20251028 08:53:45.039271 5568 raft_consensus.cc:3060] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.041461 5961 ts_tablet_manager.cc:1929] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.041625 5961 log.cc:1199] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/81d606fd4f6c4576bf885b3169b46901
I20251028 08:53:45.041947 5961 ts_tablet_manager.cc:1950] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
I20251028 08:53:45.042475 5223 catalog_manager.cc:4985] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): tablet 81d606fd4f6c4576bf885b3169b46901 (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]) successfully deleted
I20251028 08:53:45.043449 5568 raft_consensus.cc:2468] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 1.
I20251028 08:53:45.044589 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81d606fd4f6c4576bf885b3169b46901" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7" is_pre_election: true
I20251028 08:53:45.044984 5706 raft_consensus.cc:2468] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 0.
I20251028 08:53:45.045469 5639 leader_election.cc:304] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7; no voters: bbab3d08643e49e786c2174faaa741d9
I20251028 08:53:45.045876 5964 raft_consensus.cc:2804] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:45.049930 5773 leader_election.cc:304] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7, bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed
W20251028 08:53:45.050388 5773 raft_consensus.cc:2686] T 81d606fd4f6c4576bf885b3169b46901 P bbab3d08643e49e786c2174faaa741d9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251028 08:53:45.052249 5963 tablet_replica.cc:333] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:53:45.052768 5963 raft_consensus.cc:2243] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.053380 5963 raft_consensus.cc:2272] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.061062 5963 ts_tablet_manager.cc:1916] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.061285 5819 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d606fd4f6c4576bf885b3169b46901 with delete_type TABLET_DATA_DELETED (replaced by tablet 93164793833348ef97b3d6f7fe53ba8d) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:45.061524 5820 tablet_service.cc:1552] Processing DeleteTablet for tablet 4289997e6a7f459187743edfe904bd66 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c248e923a4f4085973a41a96fd1e0b7) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:45.062326 5964 raft_consensus.cc:697] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 LEADER]: Becoming Leader. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Running, Role: LEADER
W20251028 08:53:45.062701 5223 catalog_manager.cc:4952] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): delete failed for tablet 81d606fd4f6c4576bf885b3169b46901 because the tablet was not found. No further retry: Not found: Tablet not found: 81d606fd4f6c4576bf885b3169b46901
W20251028 08:53:45.062954 5223 catalog_manager.cc:4952] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): delete failed for tablet 4289997e6a7f459187743edfe904bd66 because the tablet was not found. No further retry: Not found: Tablet not found: 4289997e6a7f459187743edfe904bd66
I20251028 08:53:45.062925 5964 consensus_queue.cc:237] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.067073 5954 tablet_bootstrap.cc:492] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.067258 5954 ts_tablet_manager.cc:1403] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.148s user 0.027s sys 0.066s
I20251028 08:53:45.068190 5954 raft_consensus.cc:359] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.068406 5954 raft_consensus.cc:385] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.068504 5954 raft_consensus.cc:740] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.068749 5954 consensus_queue.cc:260] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.069456 5954 ts_tablet_manager.cc:1434] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251028 08:53:45.070272 5954 tablet_bootstrap.cc:492] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:45.070812 5229 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_c896dce32fc04719beb2467b2bf1f855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:45.072288 5954 tablet_bootstrap.cc:654] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.076787 5963 ts_tablet_manager.cc:1929] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.076941 5963 log.cc:1199] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/81d606fd4f6c4576bf885b3169b46901
I20251028 08:53:45.078482 5963 ts_tablet_manager.cc:1950] T 81d606fd4f6c4576bf885b3169b46901 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
I20251028 08:53:45.080406 5954 tablet_bootstrap.cc:492] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.080488 5222 catalog_manager.cc:4985] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): tablet 81d606fd4f6c4576bf885b3169b46901 (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]) successfully deleted
I20251028 08:53:45.080565 5954 ts_tablet_manager.cc:1403] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20251028 08:53:45.081528 5954 raft_consensus.cc:359] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.081727 5954 raft_consensus.cc:385] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.081820 5954 raft_consensus.cc:740] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.082057 5954 consensus_queue.cc:260] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.082607 5954 ts_tablet_manager.cc:1434] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251028 08:53:45.083948 5222 catalog_manager.cc:4952] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): delete failed for tablet 4289997e6a7f459187743edfe904bd66 because the tablet was not found. No further retry: Not found: Tablet not found: 4289997e6a7f459187743edfe904bd66
I20251028 08:53:45.086742 5953 tablet_bootstrap.cc:492] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.086877 5953 ts_tablet_manager.cc:1403] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.173s user 0.060s sys 0.061s
I20251028 08:53:45.087077 5955 tablet_bootstrap.cc:492] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.087234 5955 ts_tablet_manager.cc:1403] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.155s user 0.045s sys 0.053s
I20251028 08:53:45.087603 5953 raft_consensus.cc:359] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.087759 5953 raft_consensus.cc:385] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.087846 5953 raft_consensus.cc:740] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.087936 5955 raft_consensus.cc:359] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.088080 5953 consensus_queue.cc:260] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.088150 5955 raft_consensus.cc:385] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.088238 5955 raft_consensus.cc:740] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.088501 5955 consensus_queue.cc:260] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.089012 5955 ts_tablet_manager.cc:1434] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251028 08:53:45.089371 5955 tablet_bootstrap.cc:492] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.090616 5953 ts_tablet_manager.cc:1434] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251028 08:53:45.090925 5953 tablet_bootstrap.cc:492] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.092464 5953 tablet_bootstrap.cc:654] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.093210 5955 tablet_bootstrap.cc:654] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.098259 5953 tablet_bootstrap.cc:492] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.098409 5953 ts_tablet_manager.cc:1403] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20251028 08:53:45.099138 5953 raft_consensus.cc:359] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.099318 5953 raft_consensus.cc:385] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.099412 5953 raft_consensus.cc:740] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.099643 5953 consensus_queue.cc:260] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.100072 5953 ts_tablet_manager.cc:1434] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251028 08:53:45.100355 5953 tablet_bootstrap.cc:492] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.101933 5953 tablet_bootstrap.cc:654] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.103606 5955 tablet_bootstrap.cc:492] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.103734 5955 ts_tablet_manager.cc:1403] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20251028 08:53:45.104398 5955 raft_consensus.cc:359] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.104563 5955 raft_consensus.cc:385] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.104645 5955 raft_consensus.cc:740] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.104851 5955 consensus_queue.cc:260] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.107384 5953 tablet_bootstrap.cc:492] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.107443 5955 ts_tablet_manager.cc:1434] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251028 08:53:45.107501 5953 ts_tablet_manager.cc:1403] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20251028 08:53:45.108127 5953 raft_consensus.cc:359] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.108273 5953 raft_consensus.cc:385] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.108350 5953 raft_consensus.cc:740] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.108544 5953 consensus_queue.cc:260] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.109206 5953 ts_tablet_manager.cc:1434] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 08:53:45.109586 5955 tablet_bootstrap.cc:492] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.111073 5955 tablet_bootstrap.cc:654] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.122567 5955 tablet_bootstrap.cc:492] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.122706 5955 ts_tablet_manager.cc:1403] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s
I20251028 08:53:45.123337 5955 raft_consensus.cc:359] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.123530 5955 raft_consensus.cc:385] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.123600 5955 raft_consensus.cc:740] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.123791 5955 consensus_queue.cc:260] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.125907 5955 ts_tablet_manager.cc:1434] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251028 08:53:45.127951 5955 tablet_bootstrap.cc:492] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.129271 5955 tablet_bootstrap.cc:654] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.137084 5955 tablet_bootstrap.cc:492] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.137203 5955 ts_tablet_manager.cc:1403] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20251028 08:53:45.137961 5955 raft_consensus.cc:359] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.138139 5955 raft_consensus.cc:385] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.138242 5955 raft_consensus.cc:740] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.138471 5955 consensus_queue.cc:260] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.141538 5955 ts_tablet_manager.cc:1434] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251028 08:53:45.145153 5235 catalog_manager.cc:2174] created HMS catalog entry for table [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:53:45.151384 5235 catalog_manager.cc:5649] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2). New cstate: current_term: 1 leader_uuid: "52a17de1c37d4a339d1fcb35b550aad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } health_report { overall_health: UNKNOWN } } }
I20251028 08:53:45.165261 5546 tablet_service.cc:1505] Processing CreateTablet for tablet 2b26ada1ba9546e880d888184f745c40 (DEFAULT_TABLE table=default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.166232 5685 tablet_service.cc:1505] Processing CreateTablet for tablet 2b26ada1ba9546e880d888184f745c40 (DEFAULT_TABLE table=default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.166625 5685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b26ada1ba9546e880d888184f745c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.170428 5820 tablet_service.cc:1505] Processing CreateTablet for tablet 2b26ada1ba9546e880d888184f745c40 (DEFAULT_TABLE table=default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.170905 5820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b26ada1ba9546e880d888184f745c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.171304 5236 catalog_manager.cc:2174] created HMS catalog entry for table [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:53:45.173808 5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b26ada1ba9546e880d888184f745c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.186486 5236 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:45.192695 5966 raft_consensus.cc:493] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:45.192911 5954 tablet_bootstrap.cc:492] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:45.192881 5966 raft_consensus.cc:515] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.193441 5966 leader_election.cc:290] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:45.194519 5954 tablet_bootstrap.cc:654] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.194469 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ff55e6d288f44428891d6fc7e93e2d2" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.194765 5568 raft_consensus.cc:2393] T 7ff55e6d288f44428891d6fc7e93e2d2 P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbab3d08643e49e786c2174faaa741d9 in current term 1: Already voted for candidate 52a17de1c37d4a339d1fcb35b550aad7 in this term.
I20251028 08:53:45.195188 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ff55e6d288f44428891d6fc7e93e2d2" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7" is_pre_election: true
I20251028 08:53:45.196010 5953 tablet_bootstrap.cc:492] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.197439 5773 leader_election.cc:304] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:45.198071 5953 tablet_bootstrap.cc:654] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.200237 5548 tablet_service.cc:1505] Processing CreateTablet for tablet c190d47c54dd4b9fa84a1241bd4cdbcf (DEFAULT_TABLE table=default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.200618 5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c190d47c54dd4b9fa84a1241bd4cdbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.202553 5235 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40314:
table { table_name: "default.Table_dd23f4b28a514540926eb05e3f12c357" } new_table_name: "default.Table_72938f3a321a4a74ba29b7aace4a1998" modify_external_catalogs: true
I20251028 08:53:45.204644 5966 raft_consensus.cc:3060] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.207062 5966 raft_consensus.cc:2749] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:45.208681 5955 tablet_bootstrap.cc:492] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.210049 5233 catalog_manager.cc:2174] created HMS catalog entry for table [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:53:45.213148 5955 tablet_bootstrap.cc:654] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.214581 5820 tablet_service.cc:1505] Processing CreateTablet for tablet c190d47c54dd4b9fa84a1241bd4cdbcf (DEFAULT_TABLE table=default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.214917 5820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c190d47c54dd4b9fa84a1241bd4cdbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.220486 5233 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_b01e4320f600411a872c6485f421dd5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:45.225261 5685 tablet_service.cc:1505] Processing CreateTablet for tablet c190d47c54dd4b9fa84a1241bd4cdbcf (DEFAULT_TABLE table=default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.225687 5685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c190d47c54dd4b9fa84a1241bd4cdbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.226718 5684 tablet_service.cc:1505] Processing CreateTablet for tablet 2f7679e0e14447dbaf18854a5d41692a (DEFAULT_TABLE table=default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.227035 5684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7679e0e14447dbaf18854a5d41692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.229207 5954 tablet_bootstrap.cc:492] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.229424 5954 ts_tablet_manager.cc:1403] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.002s
I20251028 08:53:45.230289 5546 tablet_service.cc:1505] Processing CreateTablet for tablet 2f7679e0e14447dbaf18854a5d41692a (DEFAULT_TABLE table=default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.230479 5954 raft_consensus.cc:359] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.230742 5954 raft_consensus.cc:385] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.230724 5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7679e0e14447dbaf18854a5d41692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.230837 5954 raft_consensus.cc:740] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.231096 5954 consensus_queue.cc:260] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.235011 5954 ts_tablet_manager.cc:1434] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251028 08:53:45.236770 5231 catalog_manager.cc:2174] created HMS catalog entry for table [id=50d8e788b029409fab90accd8c14c4d2]
W20251028 08:53:45.240444 5234 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251028 08:53:45.241410 5237 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6d9d93bde7fa47158f8e7b574783370e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 08:53:45.242925 5954 tablet_bootstrap.cc:492] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:45.244498 5954 tablet_bootstrap.cc:654] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.247927 5231 master_service.cc:954] ReplaceTablet: received request to replace tablet 93164793833348ef97b3d6f7fe53ba8d from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:45.249876 5548 tablet_service.cc:1505] Processing CreateTablet for tablet 896c194b66794478b4fa1d3a08128314 (DEFAULT_TABLE table=default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.250316 5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896c194b66794478b4fa1d3a08128314. 1 dirs total, 0 dirs full, 0 dirs failed
F20251028 08:53:45.250818 5435 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f75b91d5dcd google::LogMessage::Fail() at ??:0
@ 0x7f75b91d9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f75b91d57cc google::LogMessage::Flush() at ??:0
@ 0x7f75b91d5a11 google::LogMessage::~LogMessage() at ??:0
I20251028 08:53:45.257865 5231 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Sending DeleteTablet for 3 replicas of tablet 93164793833348ef97b3d6f7fe53ba8d
I20251028 08:53:45.258333 5231 catalog_manager.cc:6719] ReplaceTablet: tablet 93164793833348ef97b3d6f7fe53ba8d deleted and replaced by tablet da51f1509b654b398d16fe14777dbbdd
I20251028 08:53:45.259995 5231 master_service.cc:954] ReplaceTablet: received request to replace tablet 4289997e6a7f459187743edfe904bd66 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:45.260515 5231 master_service.cc:954] ReplaceTablet: received request to replace tablet 633ad9f0d80c485fbe8d7bbff88f3d58 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:45.261893 5684 tablet_service.cc:1505] Processing CreateTablet for tablet 896c194b66794478b4fa1d3a08128314 (DEFAULT_TABLE table=default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.262491 5684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896c194b66794478b4fa1d3a08128314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.263578 5686 tablet_service.cc:1552] Processing DeleteTablet for tablet 93164793833348ef97b3d6f7fe53ba8d with delete_type TABLET_DATA_DELETED (replaced by tablet da51f1509b654b398d16fe14777dbbdd) from {username='slave'} at 127.0.0.1:38788
I20251028 08:53:45.263669 5548 tablet_service.cc:1552] Processing DeleteTablet for tablet 93164793833348ef97b3d6f7fe53ba8d with delete_type TABLET_DATA_DELETED (replaced by tablet da51f1509b654b398d16fe14777dbbdd) from {username='slave'} at 127.0.0.1:54274
I20251028 08:53:45.263849 5963 tablet_replica.cc:333] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:53:45.263859 5960 tablet_replica.cc:333] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
I20251028 08:53:45.264022 5960 raft_consensus.cc:2243] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.264030 5963 raft_consensus.cc:2243] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.264186 5960 raft_consensus.cc:2272] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.264202 5963 raft_consensus.cc:2272] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.264906 5960 ts_tablet_manager.cc:1916] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.265188 5963 ts_tablet_manager.cc:1916] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.270452 5954 tablet_bootstrap.cc:492] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.270591 5954 ts_tablet_manager.cc:1403] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.007s
I20251028 08:53:45.270560 5819 tablet_service.cc:1505] Processing CreateTablet for tablet 2f7679e0e14447dbaf18854a5d41692a (DEFAULT_TABLE table=default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.270905 5819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7679e0e14447dbaf18854a5d41692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.271260 5963 ts_tablet_manager.cc:1929] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.271377 5963 log.cc:1199] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/93164793833348ef97b3d6f7fe53ba8d
I20251028 08:53:45.271428 5954 raft_consensus.cc:359] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.271598 5954 raft_consensus.cc:385] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.271705 5954 raft_consensus.cc:740] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.271741 5963 ts_tablet_manager.cc:1950] T 93164793833348ef97b3d6f7fe53ba8d P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
I20251028 08:53:45.272596 5222 catalog_manager.cc:4985] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): tablet 93164793833348ef97b3d6f7fe53ba8d (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]) successfully deleted
I20251028 08:53:45.272821 5960 ts_tablet_manager.cc:1929] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.272919 5960 log.cc:1199] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/93164793833348ef97b3d6f7fe53ba8d
I20251028 08:53:45.273301 5960 ts_tablet_manager.cc:1950] T 93164793833348ef97b3d6f7fe53ba8d P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:53:45.274207 5222 catalog_manager.cc:4985] TS 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): tablet 93164793833348ef97b3d6f7fe53ba8d (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]) successfully deleted
I20251028 08:53:45.274475 5954 consensus_queue.cc:260] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.275388 5818 tablet_service.cc:1505] Processing CreateTablet for tablet 896c194b66794478b4fa1d3a08128314 (DEFAULT_TABLE table=default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.275712 5818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896c194b66794478b4fa1d3a08128314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.275878 5231 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Sending DeleteTablet for 3 replicas of tablet 633ad9f0d80c485fbe8d7bbff88f3d58
I20251028 08:53:45.276309 5231 catalog_manager.cc:6719] ReplaceTablet: tablet 633ad9f0d80c485fbe8d7bbff88f3d58 deleted and replaced by tablet 5b92cfd34a6c4e7fba8f785cc374217f
I20251028 08:53:45.276523 5231 master_service.cc:954] ReplaceTablet: received request to replace tablet 2c248e923a4f4085973a41a96fd1e0b7 from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:45.278635 5544 tablet_service.cc:1552] Processing DeleteTablet for tablet 633ad9f0d80c485fbe8d7bbff88f3d58 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b92cfd34a6c4e7fba8f785cc374217f) from {username='slave'} at 127.0.0.1:54274
I20251028 08:53:45.278673 5686 tablet_service.cc:1505] Processing CreateTablet for tablet da51f1509b654b398d16fe14777dbbdd (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.278796 5954 ts_tablet_manager.cc:1434] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251028 08:53:45.278973 5686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da51f1509b654b398d16fe14777dbbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.279024 5960 tablet_replica.cc:333] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
I20251028 08:53:45.280228 5817 tablet_service.cc:1505] Processing CreateTablet for tablet da51f1509b654b398d16fe14777dbbdd (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.280639 5817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da51f1509b654b398d16fe14777dbbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.281744 5683 tablet_service.cc:1552] Processing DeleteTablet for tablet 633ad9f0d80c485fbe8d7bbff88f3d58 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b92cfd34a6c4e7fba8f785cc374217f) from {username='slave'} at 127.0.0.1:38788
I20251028 08:53:45.281790 5960 raft_consensus.cc:2243] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.281936 5960 raft_consensus.cc:2272] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.282534 5816 tablet_service.cc:1552] Processing DeleteTablet for tablet 633ad9f0d80c485fbe8d7bbff88f3d58 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b92cfd34a6c4e7fba8f785cc374217f) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:45.282824 5961 tablet_replica.cc:333] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:53:45.282965 5961 raft_consensus.cc:2243] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.283103 5961 raft_consensus.cc:2272] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.283685 5961 ts_tablet_manager.cc:1916] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.285097 5954 tablet_bootstrap.cc:492] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:45.285378 5963 tablet_replica.cc:333] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:53:45.285563 5963 raft_consensus.cc:2243] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.285734 5963 raft_consensus.cc:2272] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.286420 5963 ts_tablet_manager.cc:1916] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.286792 5954 tablet_bootstrap.cc:654] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.287875 5544 tablet_service.cc:1505] Processing CreateTablet for tablet 5b92cfd34a6c4e7fba8f785cc374217f (DEFAULT_TABLE table=default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.288175 5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b92cfd34a6c4e7fba8f785cc374217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.292117 5963 ts_tablet_manager.cc:1929] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.292233 5963 log.cc:1199] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/633ad9f0d80c485fbe8d7bbff88f3d58
I20251028 08:53:45.292511 5963 ts_tablet_manager.cc:1950] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
I20251028 08:53:45.293359 5222 catalog_manager.cc:4985] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): tablet 633ad9f0d80c485fbe8d7bbff88f3d58 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]) successfully deleted
I20251028 08:53:45.293908 5960 ts_tablet_manager.cc:1916] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.294339 5548 tablet_service.cc:1505] Processing CreateTablet for tablet da51f1509b654b398d16fe14777dbbdd (DEFAULT_TABLE table=default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.294656 5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da51f1509b654b398d16fe14777dbbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.295007 5818 tablet_service.cc:1505] Processing CreateTablet for tablet 5b92cfd34a6c4e7fba8f785cc374217f (DEFAULT_TABLE table=default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.295287 5818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b92cfd34a6c4e7fba8f785cc374217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.297729 5819 tablet_service.cc:1552] Processing DeleteTablet for tablet 93164793833348ef97b3d6f7fe53ba8d with delete_type TABLET_DATA_DELETED (replaced by tablet da51f1509b654b398d16fe14777dbbdd) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:45.298333 5955 tablet_bootstrap.cc:492] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.298461 5955 ts_tablet_manager.cc:1403] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20251028 08:53:45.299101 5686 tablet_service.cc:1505] Processing CreateTablet for tablet 5b92cfd34a6c4e7fba8f785cc374217f (DEFAULT_TABLE table=default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.299142 5955 raft_consensus.cc:359] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.299340 5955 raft_consensus.cc:385] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.299432 5955 raft_consensus.cc:740] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.299530 5686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b92cfd34a6c4e7fba8f785cc374217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.299669 5955 consensus_queue.cc:260] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.300230 5955 ts_tablet_manager.cc:1434] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 08:53:45.300572 5955 tablet_bootstrap.cc:492] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.301410 5231 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Sending DeleteTablet for 3 replicas of tablet 2c248e923a4f4085973a41a96fd1e0b7
I20251028 08:53:45.301978 5544 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c248e923a4f4085973a41a96fd1e0b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6203a2cbeb114d37b8e4d018428c3b0d) from {username='slave'} at 127.0.0.1:54274
I20251028 08:53:45.302588 5683 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c248e923a4f4085973a41a96fd1e0b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6203a2cbeb114d37b8e4d018428c3b0d) from {username='slave'} at 127.0.0.1:38788
I20251028 08:53:45.303184 5955 tablet_bootstrap.cc:654] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
@ 0x7f75cfb97b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251028 08:53:45.309551 5231 catalog_manager.cc:6719] ReplaceTablet: tablet 2c248e923a4f4085973a41a96fd1e0b7 deleted and replaced by tablet 6203a2cbeb114d37b8e4d018428c3b0d
I20251028 08:53:45.309777 5817 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c248e923a4f4085973a41a96fd1e0b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6203a2cbeb114d37b8e4d018428c3b0d) from {username='slave'} at 127.0.0.1:56904
I20251028 08:53:45.310662 5961 ts_tablet_manager.cc:1929] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.310757 5961 log.cc:1199] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/633ad9f0d80c485fbe8d7bbff88f3d58
I20251028 08:53:45.311120 5961 ts_tablet_manager.cc:1950] T 633ad9f0d80c485fbe8d7bbff88f3d58 P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
I20251028 08:53:45.311949 5223 catalog_manager.cc:4985] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): tablet 633ad9f0d80c485fbe8d7bbff88f3d58 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]) successfully deleted
I20251028 08:53:45.312198 5961 tablet_replica.cc:333] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:53:45.312361 5961 raft_consensus.cc:2243] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.312491 5961 raft_consensus.cc:2272] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.313126 5961 ts_tablet_manager.cc:1916] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.321951 5955 tablet_bootstrap.cc:492] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.322073 5955 ts_tablet_manager.cc:1403] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20251028 08:53:45.322697 5955 raft_consensus.cc:359] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.322836 5955 raft_consensus.cc:385] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.322906 5955 raft_consensus.cc:740] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.323091 5955 consensus_queue.cc:260] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.325104 5544 tablet_service.cc:1505] Processing CreateTablet for tablet 6203a2cbeb114d37b8e4d018428c3b0d (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.325425 5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6203a2cbeb114d37b8e4d018428c3b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.325839 5955 ts_tablet_manager.cc:1434] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251028 08:53:45.327877 5961 ts_tablet_manager.cc:1929] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.327996 5961 log.cc:1199] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/93164793833348ef97b3d6f7fe53ba8d
I20251028 08:53:45.328289 5961 ts_tablet_manager.cc:1950] T 93164793833348ef97b3d6f7fe53ba8d P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
I20251028 08:53:45.328384 5955 tablet_bootstrap.cc:492] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.329098 5223 catalog_manager.cc:4985] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): tablet 93164793833348ef97b3d6f7fe53ba8d (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]) successfully deleted
I20251028 08:53:45.333000 5955 tablet_bootstrap.cc:654] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.333232 5961 tablet_replica.cc:333] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:53:45.333426 5961 raft_consensus.cc:2243] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.333585 5961 raft_consensus.cc:2272] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.334695 5683 tablet_service.cc:1505] Processing CreateTablet for tablet 6203a2cbeb114d37b8e4d018428c3b0d (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.335129 5683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6203a2cbeb114d37b8e4d018428c3b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.336143 5960 ts_tablet_manager.cc:1929] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.336297 5960 log.cc:1199] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/633ad9f0d80c485fbe8d7bbff88f3d58
I20251028 08:53:45.336680 5960 ts_tablet_manager.cc:1950] T 633ad9f0d80c485fbe8d7bbff88f3d58 P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:53:45.337531 5222 catalog_manager.cc:4985] TS 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): tablet 633ad9f0d80c485fbe8d7bbff88f3d58 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]) successfully deleted
I20251028 08:53:45.337854 5960 tablet_replica.cc:333] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
I20251028 08:53:45.337988 5960 raft_consensus.cc:2243] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.338119 5960 raft_consensus.cc:2272] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.341776 5963 tablet_replica.cc:333] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:53:45.341949 5963 raft_consensus.cc:2243] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:45.342114 5963 raft_consensus.cc:2272] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:45.343077 5820 tablet_service.cc:1505] Processing CreateTablet for tablet 6203a2cbeb114d37b8e4d018428c3b0d (DEFAULT_TABLE table=default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:45.343441 5820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6203a2cbeb114d37b8e4d018428c3b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:45.345940 5964 raft_consensus.cc:493] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:45.346073 5964 raft_consensus.cc:515] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.346657 5964 leader_election.cc:290] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.346675 5231 master_service.cc:954] ReplaceTablet: received request to replace tablet 93164793833348ef97b3d6f7fe53ba8d from {username='slave'} at 127.0.0.1:40314
I20251028 08:53:45.347276 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b26ada1ba9546e880d888184f745c40" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.347232 5231 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40314:
name: "default.Table_c496c6ebeb1d44248b341428961d8939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251028 08:53:45.347782 5639 leader_election.cc:343] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Illegal state: must be running to vote when last-logged opid is not known
I20251028 08:53:45.348349 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b26ada1ba9546e880d888184f745c40" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
I20251028 08:53:45.348573 5840 raft_consensus.cc:2468] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 0.
I20251028 08:53:45.348939 5640 leader_election.cc:304] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7, bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed
I20251028 08:53:45.349350 5964 raft_consensus.cc:2804] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:45.349440 5964 raft_consensus.cc:493] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:45.349553 5964 raft_consensus.cc:3060] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.347234 5960 ts_tablet_manager.cc:1916] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.352394 5964 raft_consensus.cc:515] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.353417 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b26ada1ba9546e880d888184f745c40" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed"
I20251028 08:53:45.353575 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b26ada1ba9546e880d888184f745c40" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9"
I20251028 08:53:45.353796 5840 raft_consensus.cc:3060] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
W20251028 08:53:45.353886 5639 leader_election.cc:343] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Illegal state: must be running to vote when last-logged opid is not known
I20251028 08:53:45.354360 5963 ts_tablet_manager.cc:1916] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.354569 5953 tablet_bootstrap.cc:492] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.354710 5953 ts_tablet_manager.cc:1403] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.159s user 0.011s sys 0.014s
I20251028 08:53:45.355403 5953 raft_consensus.cc:359] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.355612 5953 raft_consensus.cc:385] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.355685 5953 raft_consensus.cc:740] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.355742 5960 ts_tablet_manager.cc:1929] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.355835 5960 log.cc:1199] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/2c248e923a4f4085973a41a96fd1e0b7
I20251028 08:53:45.355958 5953 consensus_queue.cc:260] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.356186 5960 ts_tablet_manager.cc:1950] T 2c248e923a4f4085973a41a96fd1e0b7 P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:53:45.356600 5840 raft_consensus.cc:2468] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 1.
I20251028 08:53:45.357252 5222 catalog_manager.cc:4985] TS 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): tablet 2c248e923a4f4085973a41a96fd1e0b7 (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]) successfully deleted
I20251028 08:53:45.357923 5640 leader_election.cc:304] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7, bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed
I20251028 08:53:45.358191 5981 raft_consensus.cc:2804] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:45.358245 5964 leader_election.cc:290] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Requested vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.359586 5953 ts_tablet_manager.cc:1434] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251028 08:53:45.361150 5953 tablet_bootstrap.cc:492] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.363289 5953 tablet_bootstrap.cc:654] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.364921 5953 tablet_bootstrap.cc:492] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.365039 5953 ts_tablet_manager.cc:1403] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251028 08:53:45.365649 5953 raft_consensus.cc:359] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.365782 5953 raft_consensus.cc:385] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.365849 5953 raft_consensus.cc:740] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.365864 5981 raft_consensus.cc:697] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 LEADER]: Becoming Leader. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Running, Role: LEADER
I20251028 08:53:45.366022 5953 consensus_queue.cc:260] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.366158 5981 consensus_queue.cc:237] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.366667 5955 tablet_bootstrap.cc:492] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.366801 5955 ts_tablet_manager.cc:1403] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20251028 08:53:45.367489 5955 raft_consensus.cc:359] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.367684 5955 raft_consensus.cc:385] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.367750 5955 raft_consensus.cc:740] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.367965 5955 consensus_queue.cc:260] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.372292 5953 ts_tablet_manager.cc:1434] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251028 08:53:45.373190 5963 ts_tablet_manager.cc:1929] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.373306 5963 log.cc:1199] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/2c248e923a4f4085973a41a96fd1e0b7
I20251028 08:53:45.374833 5953 tablet_bootstrap.cc:492] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.376442 5953 tablet_bootstrap.cc:654] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.377364 5955 ts_tablet_manager.cc:1434] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
@ 0x7f75cfb96411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251028 08:53:45.380180 5963 ts_tablet_manager.cc:1950] T 2c248e923a4f4085973a41a96fd1e0b7 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
I20251028 08:53:45.381260 5222 catalog_manager.cc:4985] TS 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): tablet 2c248e923a4f4085973a41a96fd1e0b7 (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]) successfully deleted
I20251028 08:53:45.381762 5961 ts_tablet_manager.cc:1916] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:45.386622 5955 tablet_bootstrap.cc:492] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.388095 5955 tablet_bootstrap.cc:654] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.389752 5954 tablet_bootstrap.cc:492] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.389899 5954 ts_tablet_manager.cc:1403] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.022s
I20251028 08:53:45.391256 5953 tablet_bootstrap.cc:492] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.391400 5953 ts_tablet_manager.cc:1403] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251028 08:53:45.392742 5953 raft_consensus.cc:359] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.392915 5953 raft_consensus.cc:385] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.392977 5953 raft_consensus.cc:740] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.393188 5953 consensus_queue.cc:260] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:45.393793 5954 raft_consensus.cc:359] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.394479 5954 raft_consensus.cc:385] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.394552 5954 raft_consensus.cc:740] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.394966 5954 consensus_queue.cc:260] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.396013 5953 ts_tablet_manager.cc:1434] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251028 08:53:45.397222 5954 ts_tablet_manager.cc:1434] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251028 08:53:45.398959 5953 tablet_bootstrap.cc:492] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.399011 5954 tablet_bootstrap.cc:492] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:45.400519 5953 tablet_bootstrap.cc:654] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.400602 5954 tablet_bootstrap.cc:654] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.406343 5954 tablet_bootstrap.cc:492] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.406451 5954 ts_tablet_manager.cc:1403] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20251028 08:53:45.407044 5954 raft_consensus.cc:359] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.407177 5954 raft_consensus.cc:385] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.407254 5954 raft_consensus.cc:740] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.408716 5954 consensus_queue.cc:260] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.410281 5954 ts_tablet_manager.cc:1434] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251028 08:53:45.411790 5954 tablet_bootstrap.cc:492] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:45.414744 5954 tablet_bootstrap.cc:654] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.421202 5954 tablet_bootstrap.cc:492] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.421350 5954 ts_tablet_manager.cc:1403] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20251028 08:53:45.421996 5954 raft_consensus.cc:359] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.422148 5954 raft_consensus.cc:385] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.422233 5954 raft_consensus.cc:740] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.423813 5961 ts_tablet_manager.cc:1929] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251028 08:53:45.423924 5961 log.cc:1199] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/2c248e923a4f4085973a41a96fd1e0b7
I20251028 08:53:45.423883 5954 consensus_queue.cc:260] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.424271 5961 ts_tablet_manager.cc:1950] T 2c248e923a4f4085973a41a96fd1e0b7 P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
@ 0x7f75cfb9e08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251028 08:53:45.425328 5223 catalog_manager.cc:4985] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): tablet 2c248e923a4f4085973a41a96fd1e0b7 (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]) successfully deleted
I20251028 08:53:45.425451 5954 ts_tablet_manager.cc:1434] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251028 08:53:45.426975 5954 tablet_bootstrap.cc:492] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
I20251028 08:53:45.428503 5954 tablet_bootstrap.cc:654] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.431921 5955 tablet_bootstrap.cc:492] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.432044 5955 ts_tablet_manager.cc:1403] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20251028 08:53:45.432793 5955 raft_consensus.cc:359] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.432955 5955 raft_consensus.cc:385] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.433033 5955 raft_consensus.cc:740] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.433259 5955 consensus_queue.cc:260] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.434295 5954 tablet_bootstrap.cc:492] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.434427 5954 ts_tablet_manager.cc:1403] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20251028 08:53:45.435204 5955 ts_tablet_manager.cc:1434] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251028 08:53:45.436915 5954 raft_consensus.cc:359] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.437048 5954 raft_consensus.cc:385] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.437112 5954 raft_consensus.cc:740] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.438041 5955 tablet_bootstrap.cc:492] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.438072 5954 consensus_queue.cc:260] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.439487 5955 tablet_bootstrap.cc:654] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.439518 5954 ts_tablet_manager.cc:1434] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
@ 0x7f75cfb9df09 std::_Function_handler<>::_M_invoke() at ??:0
I20251028 08:53:45.444495 5998 raft_consensus.cc:493] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:45.444607 5998 raft_consensus.cc:515] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.445191 5998 leader_election.cc:290] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.445809 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6203a2cbeb114d37b8e4d018428c3b0d" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
W20251028 08:53:45.446249 5640 leader_election.cc:343] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Illegal state: must be running to vote when last-logged opid is not known
I20251028 08:53:45.446430 5953 tablet_bootstrap.cc:492] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.446578 5953 ts_tablet_manager.cc:1403] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.001s
I20251028 08:53:45.446846 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6203a2cbeb114d37b8e4d018428c3b0d" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.447292 5953 raft_consensus.cc:359] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
W20251028 08:53:45.447346 5639 leader_election.cc:343] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Illegal state: must be running to vote when last-logged opid is not known
I20251028 08:53:45.447443 5953 raft_consensus.cc:385] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.447515 5953 raft_consensus.cc:740] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f75bbf49ee9 kudu::Thread::SuperviseThread() at ??:0
I20251028 08:53:45.454579 5955 tablet_bootstrap.cc:492] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.473165 5955 ts_tablet_manager.cc:1403] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.001s
I20251028 08:53:45.447449 5639 leader_election.cc:304] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7; no voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9
I20251028 08:53:45.473914 5955 raft_consensus.cc:359] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.447734 5953 consensus_queue.cc:260] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.474085 5955 raft_consensus.cc:385] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.474149 5955 raft_consensus.cc:740] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.474375 5955 consensus_queue.cc:260] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.474476 5953 ts_tablet_manager.cc:1434] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20251028 08:53:45.474766 5953 tablet_bootstrap.cc:492] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.474782 5955 ts_tablet_manager.cc:1434] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 08:53:45.475029 5955 tablet_bootstrap.cc:492] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
@ 0x7f75bf5546db start_thread at ??:0
I20251028 08:53:45.476357 5953 tablet_bootstrap.cc:654] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.476425 5955 tablet_bootstrap.cc:654] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.452106 5954 tablet_bootstrap.cc:492] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: Bootstrap starting.
@ 0x7f75b808671f clone at ??:0
I20251028 08:53:45.478519 5954 tablet_bootstrap.cc:654] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.479338 5981 raft_consensus.cc:2749] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:45.488480 5954 tablet_bootstrap.cc:492] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: No bootstrap required, opened a new log
I20251028 08:53:45.488641 5954 ts_tablet_manager.cc:1403] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20251028 08:53:45.499825 5954 raft_consensus.cc:359] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.499994 5954 raft_consensus.cc:385] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.500052 5954 raft_consensus.cc:740] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.493714 5953 tablet_bootstrap.cc:492] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.500254 5953 ts_tablet_manager.cc:1403] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.006s
I20251028 08:53:45.500305 5954 consensus_queue.cc:260] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.500769 5954 ts_tablet_manager.cc:1434] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 08:53:45.500975 5953 raft_consensus.cc:359] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.501156 5953 raft_consensus.cc:385] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.501223 5953 raft_consensus.cc:740] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.501434 5953 consensus_queue.cc:260] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.501559 5955 tablet_bootstrap.cc:492] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.501688 5955 ts_tablet_manager.cc:1403] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20251028 08:53:45.501901 5953 ts_tablet_manager.cc:1434] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 08:53:45.502151 5953 tablet_bootstrap.cc:492] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.502327 5955 raft_consensus.cc:359] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.502460 5955 raft_consensus.cc:385] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.502526 5955 raft_consensus.cc:740] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.502700 5955 consensus_queue.cc:260] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.503134 5955 ts_tablet_manager.cc:1434] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251028 08:53:45.503379 5955 tablet_bootstrap.cc:492] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:45.506654 5955 tablet_bootstrap.cc:654] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.509279 5953 tablet_bootstrap.cc:654] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.521941 5953 tablet_bootstrap.cc:492] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.522099 5953 ts_tablet_manager.cc:1403] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.003s
I20251028 08:53:45.522735 5953 raft_consensus.cc:359] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.522879 5953 raft_consensus.cc:385] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.522993 5953 raft_consensus.cc:740] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.523218 5953 consensus_queue.cc:260] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.527276 5953 ts_tablet_manager.cc:1434] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251028 08:53:45.530115 5966 raft_consensus.cc:493] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:45.530248 5966 raft_consensus.cc:515] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.530857 5966 leader_election.cc:290] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:45.531558 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c190d47c54dd4b9fa84a1241bd4cdbcf" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7" is_pre_election: true
I20251028 08:53:45.531759 5706 raft_consensus.cc:2468] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 0.
I20251028 08:53:45.532160 5773 leader_election.cc:304] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:45.532794 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c190d47c54dd4b9fa84a1241bd4cdbcf" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.532958 5568 raft_consensus.cc:2468] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 0.
I20251028 08:53:45.533195 5966 raft_consensus.cc:2804] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:45.533304 5966 raft_consensus.cc:493] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:45.533385 5966 raft_consensus.cc:3060] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.536876 5966 raft_consensus.cc:515] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.537940 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c190d47c54dd4b9fa84a1241bd4cdbcf" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed"
I20251028 08:53:45.538092 5568 raft_consensus.cc:3060] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.540761 5568 raft_consensus.cc:2468] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 1.
I20251028 08:53:45.541141 5773 leader_election.cc:304] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:45.541792 5966 leader_election.cc:290] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 election: Requested vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:45.541999 5966 raft_consensus.cc:2804] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:45.542122 5966 raft_consensus.cc:697] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 1 LEADER]: Becoming Leader. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Running, Role: LEADER
I20251028 08:53:45.542481 5966 consensus_queue.cc:237] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.550043 5953 tablet_bootstrap.cc:492] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: Bootstrap starting.
I20251028 08:53:45.559216 5953 tablet_bootstrap.cc:654] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:45.568127 5955 tablet_bootstrap.cc:492] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:45.568298 5955 ts_tablet_manager.cc:1403] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.013s
I20251028 08:53:45.568969 5953 tablet_bootstrap.cc:492] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: No bootstrap required, opened a new log
I20251028 08:53:45.569099 5953 ts_tablet_manager.cc:1403] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20251028 08:53:45.569031 5955 raft_consensus.cc:359] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.569185 5955 raft_consensus.cc:385] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.569247 5955 raft_consensus.cc:740] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.570466 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c190d47c54dd4b9fa84a1241bd4cdbcf" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7"
I20251028 08:53:45.570688 5706 raft_consensus.cc:3060] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.573478 5706 raft_consensus.cc:2468] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 1.
I20251028 08:53:45.573803 5955 consensus_queue.cc:260] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.574337 5955 ts_tablet_manager.cc:1434] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251028 08:53:45.581791 5953 raft_consensus.cc:359] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.583048 5953 raft_consensus.cc:385] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:45.583122 5953 raft_consensus.cc:740] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.583361 5953 consensus_queue.cc:260] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.586685 5953 ts_tablet_manager.cc:1434] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251028 08:53:45.627837 5981 raft_consensus.cc:493] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:45.627993 5981 raft_consensus.cc:515] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.628651 5981 leader_election.cc:290] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.629390 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b92cfd34a6c4e7fba8f785cc374217f" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
I20251028 08:53:45.629647 5840 raft_consensus.cc:2468] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 0.
I20251028 08:53:45.630095 5640 leader_election.cc:304] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:45.630359 5981 raft_consensus.cc:2804] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:45.630463 5981 raft_consensus.cc:493] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:45.630571 5981 raft_consensus.cc:3060] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.633140 5981 raft_consensus.cc:515] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.634212 5568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b92cfd34a6c4e7fba8f785cc374217f" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.634320 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b92cfd34a6c4e7fba8f785cc374217f" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9"
I20251028 08:53:45.634403 5568 raft_consensus.cc:2468] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 0.
I20251028 08:53:45.634493 5840 raft_consensus.cc:3060] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.634720 5567 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b92cfd34a6c4e7fba8f785cc374217f" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed"
I20251028 08:53:45.634908 5567 raft_consensus.cc:3060] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.637220 5840 raft_consensus.cc:2468] T 5b92cfd34a6c4e7fba8f785cc374217f P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 1.
I20251028 08:53:45.637544 5640 leader_election.cc:304] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:45.637712 5567 raft_consensus.cc:2468] T 5b92cfd34a6c4e7fba8f785cc374217f P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 1.
I20251028 08:53:45.637818 5998 raft_consensus.cc:2804] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:45.637960 5998 raft_consensus.cc:697] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 LEADER]: Becoming Leader. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Running, Role: LEADER
I20251028 08:53:45.638201 5998 consensus_queue.cc:237] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.640641 5981 leader_election.cc:290] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Requested vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
08:53:45.666 [HMS - ERROR - pool-6-thread-1] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I20251028 08:53:45.667958 5998 raft_consensus.cc:493] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:45.668107 5998 raft_consensus.cc:515] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.668747 5998 leader_election.cc:290] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.669109 5567 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da51f1509b654b398d16fe14777dbbdd" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:45.669239 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da51f1509b654b398d16fe14777dbbdd" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
I20251028 08:53:45.669310 5567 raft_consensus.cc:2468] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 0.
I20251028 08:53:45.669376 5840 raft_consensus.cc:2468] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 0.
I20251028 08:53:45.669808 5639 leader_election.cc:304] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7; no voters:
I20251028 08:53:45.670176 5998 raft_consensus.cc:2804] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:45.670277 5998 raft_consensus.cc:493] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:45.670384 5998 raft_consensus.cc:3060] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.672205 5998 raft_consensus.cc:515] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:45.672940 5998 leader_election.cc:290] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Requested vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.672976 5567 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da51f1509b654b398d16fe14777dbbdd" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed"
I20251028 08:53:45.673038 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da51f1509b654b398d16fe14777dbbdd" candidate_uuid: "52a17de1c37d4a339d1fcb35b550aad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9"
I20251028 08:53:45.673146 5567 raft_consensus.cc:3060] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.673182 5840 raft_consensus.cc:3060] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.675277 5840 raft_consensus.cc:2468] T da51f1509b654b398d16fe14777dbbdd P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 1.
I20251028 08:53:45.675657 5640 leader_election.cc:304] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a17de1c37d4a339d1fcb35b550aad7, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:45.675771 5567 raft_consensus.cc:2468] T da51f1509b654b398d16fe14777dbbdd P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a17de1c37d4a339d1fcb35b550aad7 in term 1.
I20251028 08:53:45.676437 5998 raft_consensus.cc:2804] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:45.676899 5998 raft_consensus.cc:697] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 LEADER]: Becoming Leader. State: Replica: 52a17de1c37d4a339d1fcb35b550aad7, State: Running, Role: LEADER
I20251028 08:53:45.677489 5998 consensus_queue.cc:237] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
W20251028 08:53:45.680851 5980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:45.681183 5980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:45.681293 5980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:45.690156 5980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:53:45.690227 5980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:45.690284 5980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:53:45.690348 5980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:53:45.699477 5980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:45.700984 5980 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:45.702672 5980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:45.710506 6034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:45.710531 6033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:45.711385 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:45.711817 5980 server_base.cc:1047] running on GCE node
I20251028 08:53:45.712656 5980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:45.714148 5980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:45.715389 5980 hybrid_clock.cc:648] HybridClock initialized: now 1761641625715331 us; error 40 us; skew 500 ppm
I20251028 08:53:45.715775 5980 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:45.718331 5980 webserver.cc:492] Webserver started at http://127.4.217.60:41817/ using document root <none> and password file <none>
I20251028 08:53:45.719194 5980 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:45.719331 5980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:45.725571 5980 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20251028 08:53:45.728663 6042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:45.730084 5980 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251028 08:53:45.730258 5980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
uuid: "82b026d06b454d65b27216e3002f29af"
format_stamp: "Formatted at 2025-10-28 08:53:42 on dist-test-slave-cmml"
I20251028 08:53:45.731148 5980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:45.760044 5980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:45.760946 5980 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:45.761248 5980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:45.786206 5980 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:37319
I20251028 08:53:45.787009 6093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.60:37319 every 8 connection(s)
I20251028 08:53:45.787992 5980 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
I20251028 08:53:45.794369 6094 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:53:45.795876 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 5980
I20251028 08:53:45.797827 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap starting.
I20251028 08:53:45.809720 6094 log.cc:826] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Log is configured to *not* fsync() on all Append() calls
W20251028 08:53:45.811642 5894 connection.cc:537] client connection to 127.4.217.62:41357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:45.812465 5894 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.217.62:41357)
W20251028 08:53:45.812920 5894 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.62:41357)
W20251028 08:53:45.813469 5894 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.217.62:41357)
W20251028 08:53:45.814062 5894 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.217.62:41357)
W20251028 08:53:45.816423 5970 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 08:53:45.815328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41357 (local address 127.4.217.1:57763)
1028 08:53:45.815527 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1028 08:53:45.815541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1028 08:53:45.815648 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1028 08:53:45.815775 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":101}
W20251028 08:53:45.817209 5978 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 08:53:45.812298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41357 (local address 127.4.217.3:57843)
1028 08:53:45.812440 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1028 08:53:45.812456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1028 08:53:45.812562 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1028 08:53:45.816689 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20251028 08:53:45.818110 5977 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 08:53:45.813216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.62:41357 (local address 127.4.217.2:41625)
1028 08:53:45.813409 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1028 08:53:45.813428 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1028 08:53:45.813541 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1028 08:53:45.817554 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
I20251028 08:53:45.824457 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 08:53:45.824985 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap complete.
I20251028 08:53:45.833743 6094 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:45.834860 6094 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b026d06b454d65b27216e3002f29af, State: Initialized, Role: FOLLOWER
I20251028 08:53:45.835610 6094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:45.836469 6101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:45.836850 6101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:45.838163 6094 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:53:45.844707 6100 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0b9421be172b482f9c69d4dc5d2b19a8)
I20251028 08:53:45.844843 6100 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:45.853729 6100 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:53:45.865233 5998 consensus_queue.cc:1048] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:45.869243 6115 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:45.869316 6115 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: loading cluster ID for follower catalog manager: success
I20251028 08:53:45.873232 6115 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: acquiring CA information for follower catalog manager: success
I20251028 08:53:45.875568 6115 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:53:45.875819 5980 master_runner.cc:186] Error getting master registration for 127.4.217.62:41357: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111),
I20251028 08:53:45.884891 5980 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:53:45.886207 5885 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:45.886021 6069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af" is_pre_election: true
I20251028 08:53:45.886385 5885 heartbeater.cc:499] Master 127.4.217.60:37319 was elected leader, sending a full tablet report...
I20251028 08:53:45.886636 6069 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 1.
I20251028 08:53:45.887226 5292 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7; no voters:
I20251028 08:53:45.887554 6100 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251028 08:53:45.887673 6100 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0b9421be172b482f9c69d4dc5d2b19a8)
I20251028 08:53:45.887763 6100 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 2 FOLLOWER]: Advancing to term 3
I20251028 08:53:45.890825 6100 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:45.891695 6100 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 3 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:53:45.892179 6069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af"
I20251028 08:53:45.892426 6069 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 1 FOLLOWER]: Advancing to term 3
I20251028 08:53:45.898952 6069 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 3.
I20251028 08:53:45.901080 5292 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7; no voters:
I20251028 08:53:45.901446 6100 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Leader election won for term 3
W20251028 08:53:45.904455 5293 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:45.905627 5293 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:53:45.908046 6059 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:44063; Asking this server to re-register.
I20251028 08:53:45.907824 6100 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 LEADER]: Becoming Leader. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Running, Role: LEADER
I20251028 08:53:45.908849 5885 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:45.908998 6100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:45.910367 5998 consensus_queue.cc:1048] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:45.911401 6058 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:45.913508 6124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 3 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:45.913815 6124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:45.919791 5751 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:45.919996 5751 heartbeater.cc:499] Master 127.4.217.60:37319 was elected leader, sending a full tablet report...
I20251028 08:53:45.922361 5614 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:45.922969 6059 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:57001; Asking this server to re-register.
I20251028 08:53:45.923035 6129 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:53:45.923624 5751 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:45.926564 6058 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:45.927410 5959 raft_consensus.cc:493] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:45.927593 5959 raft_consensus.cc:515] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.929158 5959 leader_election.cc:290] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:45.930819 6129 catalog_manager.cc:679] Loaded metadata for table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:53:45.931424 6129 catalog_manager.cc:679] Loaded metadata for table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]
I20251028 08:53:45.931949 6129 catalog_manager.cc:679] Loaded metadata for table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:53:45.932421 6129 catalog_manager.cc:679] Loaded metadata for table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:53:45.932935 6129 catalog_manager.cc:679] Loaded metadata for table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:53:45.933467 6129 catalog_manager.cc:679] Loaded metadata for table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:53:45.934010 6129 catalog_manager.cc:679] Loaded metadata for table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:53:45.934546 6129 catalog_manager.cc:679] Loaded metadata for table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:53:45.936795 6129 tablet_loader.cc:96] loaded metadata for tablet 2b26ada1ba9546e880d888184f745c40 (table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14])
I20251028 08:53:45.937724 6129 tablet_loader.cc:96] loaded metadata for tablet 2f7679e0e14447dbaf18854a5d41692a (table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3])
I20251028 08:53:45.938457 6129 tablet_loader.cc:96] loaded metadata for tablet 5b92cfd34a6c4e7fba8f785cc374217f (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad])
I20251028 08:53:45.938917 6129 tablet_loader.cc:96] loaded metadata for tablet 6203a2cbeb114d37b8e4d018428c3b0d (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4])
I20251028 08:53:45.939661 6129 tablet_loader.cc:96] loaded metadata for tablet 7ff55e6d288f44428891d6fc7e93e2d2 (table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196])
I20251028 08:53:45.940522 6129 tablet_loader.cc:96] loaded metadata for tablet 896c194b66794478b4fa1d3a08128314 (table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2])
I20251028 08:53:45.941385 6129 tablet_loader.cc:96] loaded metadata for tablet c190d47c54dd4b9fa84a1241bd4cdbcf (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e])
I20251028 08:53:45.941836 6129 tablet_loader.cc:96] loaded metadata for tablet da51f1509b654b398d16fe14777dbbdd (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508])
I20251028 08:53:45.942149 6129 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:53:45.943464 6129 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:45.943571 6129 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:53:45.945395 6129 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:53:45.946475 6059 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:45565; Asking this server to re-register.
I20251028 08:53:45.946949 6129 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Loaded TSK: 0
I20251028 08:53:45.948110 6129 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:53:45.948377 5614 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:45.949276 6129 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:53:45.950243 6059 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:45.950958 6129 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:53:45.957365 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896c194b66794478b4fa1d3a08128314" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7" is_pre_election: true
I20251028 08:53:45.957554 5706 raft_consensus.cc:2468] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 0.
I20251028 08:53:45.958020 5501 leader_election.cc:304] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7; no voters:
I20251028 08:53:45.958129 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896c194b66794478b4fa1d3a08128314" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
I20251028 08:53:45.958298 5959 raft_consensus.cc:2804] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:45.958365 5840 raft_consensus.cc:2468] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 0.
I20251028 08:53:45.958407 5959 raft_consensus.cc:493] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:45.958480 5959 raft_consensus.cc:3060] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.960534 5959 raft_consensus.cc:515] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.961066 5959 leader_election.cc:290] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 election: Requested vote from peers bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:45.961544 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896c194b66794478b4fa1d3a08128314" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9"
I20251028 08:53:45.961767 5840 raft_consensus.cc:3060] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.961727 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896c194b66794478b4fa1d3a08128314" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7"
I20251028 08:53:45.961884 5706 raft_consensus.cc:3060] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:45.962803 5308 master_service.cc:954] ReplaceTablet: received request to replace tablet 896c194b66794478b4fa1d3a08128314 from {username='slave'} at 127.0.0.1:47764
I20251028 08:53:45.964705 5840 raft_consensus.cc:2468] T 896c194b66794478b4fa1d3a08128314 P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 1.
I20251028 08:53:45.964717 5706 raft_consensus.cc:2468] T 896c194b66794478b4fa1d3a08128314 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 1.
I20251028 08:53:45.965019 5307 master_service.cc:954] ReplaceTablet: received request to replace tablet 896c194b66794478b4fa1d3a08128314 from {username='slave'} at 127.0.0.1:47764
I20251028 08:53:45.965373 5502 leader_election.cc:304] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:45.965651 5959 raft_consensus.cc:2804] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:45.967476 5959 raft_consensus.cc:697] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [term 1 LEADER]: Becoming Leader. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Running, Role: LEADER
I20251028 08:53:45.967906 5959 consensus_queue.cc:237] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:45.968221 5306 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f7679e0e14447dbaf18854a5d41692a from {username='slave'} at 127.0.0.1:47764
I20251028 08:53:45.969833 5305 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b92cfd34a6c4e7fba8f785cc374217f from {username='slave'} at 127.0.0.1:47764
I20251028 08:53:45.968989 5299 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47764:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:53:45.974934 5299 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47764:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:53:45.969254 5300 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47764:
table { table_name: "default.Table_78fd665055cb412390a37c5a185dec85" } modify_external_catalogs: true
I20251028 08:53:45.975487 5300 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47764:
table { table_name: "default.Table_78fd665055cb412390a37c5a185dec85" } modify_external_catalogs: true
I20251028 08:53:45.970983 5304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47764:
name: "default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:45.970670 5303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47764:
name: "default.Table_80a36344c9054a9e996f831074abfc41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:45.970325 5302 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47764:
name: "default.Table_541b7acce93a4f9785758f2d9acc4406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:45.969866 5301 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47764:
name: "default.Table_c22aee04efca4315947c85505204085f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251028 08:53:45.985461 5482 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f4a53b3bdcd google::LogMessage::Fail() at ??:0
I20251028 08:53:45.989380 6069 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Refusing update from remote peer 8b7d4120e0c84998858b0bd4ed9572b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
W20251028 08:53:45.989959 5293 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 -> Peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Couldn't send request to peer 0b9421be172b482f9c69d4dc5d2b19a8. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251028 08:53:45.990345 6100 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f4a53b3fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4a53b3b7cc google::LogMessage::Flush() at ??:0
@ 0x7f4a53b3ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f4a6a4fdb64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f4a6a4fc411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f4a6a50408c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f4a6a503f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f4a568afee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4a59eba6db start_thread at ??:0
@ 0x7f4a529ec71f clone at ??:0
I20251028 08:53:46.038829 6101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 3 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:46.038837 6141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:46.039094 6101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:46.039119 6141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:46.077379 6138 mvcc.cc:204] Tried to move back new op lower bound from 7215684097301393408 to 7215684092739858432. Current Snapshot: MvccSnapshot[applied={T|T < 7215684097301393408 or (T in {7215684097301393408})}]
I20251028 08:53:46.135630 5567 raft_consensus.cc:3060] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:46.137477 5959 raft_consensus.cc:493] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:46.137656 5959 raft_consensus.cc:515] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:46.138242 5959 leader_election.cc:290] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:46.139065 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6203a2cbeb114d37b8e4d018428c3b0d" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9" is_pre_election: true
I20251028 08:53:46.139269 5840 raft_consensus.cc:2468] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 0.
I20251028 08:53:46.139684 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6203a2cbeb114d37b8e4d018428c3b0d" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7" is_pre_election: true
I20251028 08:53:46.139844 5502 leader_election.cc:304] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:46.139923 5706 raft_consensus.cc:2468] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 0.
I20251028 08:53:46.140146 5998 consensus_queue.cc:1048] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.140650 5959 raft_consensus.cc:2804] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:46.140739 5959 raft_consensus.cc:493] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:46.140841 5959 raft_consensus.cc:3060] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:46.143708 5959 raft_consensus.cc:515] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:46.144296 5959 leader_election.cc:290] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 election: Requested vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:46.144721 5840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6203a2cbeb114d37b8e4d018428c3b0d" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbab3d08643e49e786c2174faaa741d9"
I20251028 08:53:46.144876 5840 raft_consensus.cc:3060] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:46.146312 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6203a2cbeb114d37b8e4d018428c3b0d" candidate_uuid: "418c8c132d8141ae96893d82e0217aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7"
I20251028 08:53:46.146468 5706 raft_consensus.cc:3060] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:46.147462 5840 raft_consensus.cc:2468] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 1.
I20251028 08:53:46.149165 5706 raft_consensus.cc:2468] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418c8c132d8141ae96893d82e0217aed in term 1.
I20251028 08:53:46.149607 5502 leader_election.cc:304] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:46.149869 5959 raft_consensus.cc:2804] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:46.149997 5959 raft_consensus.cc:697] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 1 LEADER]: Becoming Leader. State: Replica: 418c8c132d8141ae96893d82e0217aed, State: Running, Role: LEADER
I20251028 08:53:46.150249 5959 consensus_queue.cc:237] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:46.154877 5981 consensus_queue.cc:1048] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.175714 6018 raft_consensus.cc:493] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:46.175863 6018 raft_consensus.cc:515] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:46.176514 6018 leader_election.cc:290] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:46.176956 5567 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f7679e0e14447dbaf18854a5d41692a" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed" is_pre_election: true
I20251028 08:53:46.177124 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f7679e0e14447dbaf18854a5d41692a" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7" is_pre_election: true
I20251028 08:53:46.177187 5567 raft_consensus.cc:2468] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 0.
I20251028 08:53:46.177299 5706 raft_consensus.cc:2468] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 0.
I20251028 08:53:46.177714 5773 leader_election.cc:304] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:46.177951 6018 raft_consensus.cc:2804] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 08:53:46.178113 6018 raft_consensus.cc:493] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:46.178191 6018 raft_consensus.cc:3060] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:46.181094 6018 raft_consensus.cc:515] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
I20251028 08:53:46.181707 6018 leader_election.cc:290] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 election: Requested vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:46.182029 5567 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f7679e0e14447dbaf18854a5d41692a" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418c8c132d8141ae96893d82e0217aed"
I20251028 08:53:46.182229 5567 raft_consensus.cc:3060] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:46.182209 5706 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f7679e0e14447dbaf18854a5d41692a" candidate_uuid: "bbab3d08643e49e786c2174faaa741d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a17de1c37d4a339d1fcb35b550aad7"
I20251028 08:53:46.182360 5706 raft_consensus.cc:3060] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 08:53:46.184758 5567 raft_consensus.cc:2468] T 2f7679e0e14447dbaf18854a5d41692a P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 1.
I20251028 08:53:46.184906 5706 raft_consensus.cc:2468] T 2f7679e0e14447dbaf18854a5d41692a P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbab3d08643e49e786c2174faaa741d9 in term 1.
I20251028 08:53:46.185107 5773 leader_election.cc:304] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418c8c132d8141ae96893d82e0217aed, bbab3d08643e49e786c2174faaa741d9; no voters:
I20251028 08:53:46.185446 6018 raft_consensus.cc:2804] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 08:53:46.185910 6018 raft_consensus.cc:697] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [term 1 LEADER]: Becoming Leader. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Running, Role: LEADER
I20251028 08:53:46.186208 6018 consensus_queue.cc:237] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } }
W20251028 08:53:46.217157 5775 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:46.217172 5641 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:46.219987 6046 connection.cc:537] server connection from 127.0.0.1:40978 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:46.221820 6123 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 08:53:46.217884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:33257 (local address 127.4.217.3:56233)
1028 08:53:46.218342 (+ 458us) negotiation.cc:107] Waiting for socket to connect
1028 08:53:46.218361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1028 08:53:46.218487 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1028 08:53:46.221594 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":330}
W20251028 08:53:46.222374 6098 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 08:53:46.219860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:33257 (local address 127.0.0.1:47772)
1028 08:53:46.220469 (+ 609us) negotiation.cc:107] Waiting for socket to connect
1028 08:53:46.220487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1028 08:53:46.220588 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1028 08:53:46.220887 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":477}
I20251028 08:53:46.489109 6018 consensus_queue.cc:1048] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.493952 6141 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:53:46.494122 6141 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:46.495134 6162 consensus_queue.cc:1048] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.495939 6141 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:46.508356 6045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:53:46.510787 6045 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:53:46.510985 6044 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:53:46.511111 6044 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:46.511437 6141 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251028 08:53:46.628249 5981 consensus_queue.cc:1048] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.632789 5998 consensus_queue.cc:1048] T 5b92cfd34a6c4e7fba8f785cc374217f P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.643474 5998 consensus_queue.cc:1048] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.647048 5981 consensus_queue.cc:1048] T da51f1509b654b398d16fe14777dbbdd P 52a17de1c37d4a339d1fcb35b550aad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.930828 5959 consensus_queue.cc:1048] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.937091 5959 consensus_queue.cc:1048] T 896c194b66794478b4fa1d3a08128314 P 418c8c132d8141ae96893d82e0217aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.952616 5959 consensus_queue.cc:1048] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:46.956959 5959 consensus_queue.cc:1048] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:47.027010 6162 consensus_queue.cc:1048] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:47.033524 6180 consensus_queue.cc:1048] T 2f7679e0e14447dbaf18854a5d41692a P bbab3d08643e49e786c2174faaa741d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 08:53:47.091799 6183 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:53:47.091913 6183 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:47.092428 6183 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:47.094101 6045 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:53:47.094251 6044 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:53:47.094362 6044 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:47.094569 6183 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251028 08:53:47.730993 6190 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:53:47.731122 6190 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:47.731595 6190 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:47.733193 6044 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:47.733355 6045 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:47.733479 6045 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:47.733727 6190 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251028 08:53:48.980033 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--webserver_interface=127.4.217.61
--webserver_port=43565
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251028 08:53:49.042647 6209 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:53:49.042806 6209 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:49.043452 6209 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:49.045648 6044 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:49.045789 6045 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:49.045903 6045 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:49.046140 6209 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251028 08:53:49.275264 6208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:49.275496 6208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:49.275552 6208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:49.282430 6208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:53:49.282492 6208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:49.282538 6208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:53:49.282578 6208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:53:49.291111 6208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:49.292274 6208 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:49.293715 6208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:49.300575 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:49.300719 6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:49.300631 6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:49.301481 6208 server_base.cc:1047] running on GCE node
I20251028 08:53:49.302145 6208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:49.303220 6208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:49.304412 6208 hybrid_clock.cc:648] HybridClock initialized: now 1761641629304367 us; error 35 us; skew 500 ppm
I20251028 08:53:49.304687 6208 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:49.306537 6208 webserver.cc:492] Webserver started at http://127.4.217.61:43565/ using document root <none> and password file <none>
I20251028 08:53:49.307160 6208 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:49.307273 6208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:49.312193 6208 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251028 08:53:49.314560 6228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:49.315661 6208 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 08:53:49.315793 6208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:53:49.316344 6208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:49.337796 6208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:49.338392 6208 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:49.338608 6208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:49.356711 6208 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:33257
I20251028 08:53:49.356748 6279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.61:33257 every 8 connection(s)
I20251028 08:53:49.358036 6208 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
I20251028 08:53:49.363965 6280 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:53:49.365289 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 6208
I20251028 08:53:49.367492 6280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap starting.
I20251028 08:53:49.377501 6280 log.cc:826] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:49.389386 5887 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:53:49.407445 6280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20251028 08:53:49.407899 6280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap complete.
I20251028 08:53:49.412348 6280 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:49.414811 6280 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Initialized, Role: FOLLOWER
I20251028 08:53:49.415354 6280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:49.415980 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:49.416177 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:49.416790 6280 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:53:49.426821 6301 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:49.426950 6301 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: loading cluster ID for follower catalog manager: success
I20251028 08:53:49.424455 6208 master_runner.cc:186] Error getting master registration for 127.4.217.62:41357: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111),
I20251028 08:53:49.429415 6208 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:53:49.430716 6301 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: acquiring CA information for follower catalog manager: success
I20251028 08:53:49.433147 6301 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:53:49.650861 5753 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:53:49.653148 6245 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:37087; Asking this server to re-register.
I20251028 08:53:49.653738 5753 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:49.655189 6245 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:49.723232 5616 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:53:49.725020 6245 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:37743; Asking this server to re-register.
I20251028 08:53:49.725669 5616 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:49.728116 6244 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:49.894449 6244 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:40605; Asking this server to re-register.
I20251028 08:53:49.894968 5887 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:49.896034 6244 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:50.004551 6311 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:50.004758 6311 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:50.006361 6311 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:53:50.007144 6230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:50.008570 6230 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:53:50.011991 6069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af" is_pre_election: true
I20251028 08:53:50.012173 6069 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 3.
I20251028 08:53:50.012734 6229 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 0b9421be172b482f9c69d4dc5d2b19a8
I20251028 08:53:50.013074 6311 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251028 08:53:50.013196 6311 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:53:50.013316 6311 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 3 FOLLOWER]: Advancing to term 4
I20251028 08:53:50.016276 6311 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:50.016820 6311 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 4 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:53:50.017218 6069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af"
I20251028 08:53:50.017376 6069 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 3 FOLLOWER]: Advancing to term 4
W20251028 08:53:50.017879 6230 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:53:50.019289 6069 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 4.
I20251028 08:53:50.019742 6229 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 0b9421be172b482f9c69d4dc5d2b19a8
I20251028 08:53:50.020015 6311 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Leader election won for term 4
I20251028 08:53:50.020376 6311 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 LEADER]: Becoming Leader. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Running, Role: LEADER
I20251028 08:53:50.020756 6311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.36, Last appended by leader: 36, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:50.023241 6316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 4 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:50.023536 6316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
W20251028 08:53:50.516480 6230 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 -> Peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Couldn't send request to peer 0b9421be172b482f9c69d4dc5d2b19a8. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251028 08:53:50.583691 6069 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Refusing update from remote peer 8b7d4120e0c84998858b0bd4ed9572b7: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20251028 08:53:50.584357 6316 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
I20251028 08:53:50.588016 6321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 4 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:50.588227 6321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:50.593061 6321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:50.593485 6316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:50.593729 6316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:50.593554 6332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:50.593971 6332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:53:50.593717 6331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:50.594165 6331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:50.594308 6321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:50.668915 6318 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:53:50.671231 6318 catalog_manager.cc:679] Loaded metadata for table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:53:50.671665 6318 catalog_manager.cc:679] Loaded metadata for table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]
I20251028 08:53:50.672014 6318 catalog_manager.cc:679] Loaded metadata for table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:53:50.672330 6318 catalog_manager.cc:679] Loaded metadata for table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:53:50.672649 6318 catalog_manager.cc:679] Loaded metadata for table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:53:50.672967 6318 catalog_manager.cc:679] Loaded metadata for table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:53:50.673285 6318 catalog_manager.cc:679] Loaded metadata for table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:53:50.673592 6318 catalog_manager.cc:679] Loaded metadata for table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:53:50.675019 6318 tablet_loader.cc:96] loaded metadata for tablet 2b26ada1ba9546e880d888184f745c40 (table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14])
I20251028 08:53:50.675933 6318 tablet_loader.cc:96] loaded metadata for tablet 6203a2cbeb114d37b8e4d018428c3b0d (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4])
I20251028 08:53:50.676326 6318 tablet_loader.cc:96] loaded metadata for tablet 774e1d1fe9334a45a7df75cc5cd098ff (table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2])
I20251028 08:53:50.676565 6318 tablet_loader.cc:96] loaded metadata for tablet 7a5654d71877406f8a5a4e1e9bb08fa4 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad])
I20251028 08:53:50.676831 6318 tablet_loader.cc:96] loaded metadata for tablet 7ff55e6d288f44428891d6fc7e93e2d2 (table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196])
I20251028 08:53:50.677523 6318 tablet_loader.cc:96] loaded metadata for tablet c190d47c54dd4b9fa84a1241bd4cdbcf (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e])
I20251028 08:53:50.677809 6318 tablet_loader.cc:96] loaded metadata for tablet da51f1509b654b398d16fe14777dbbdd (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508])
I20251028 08:53:50.678087 6318 tablet_loader.cc:96] loaded metadata for tablet e591ce42e8ce4df0952cb231526544ab (table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3])
I20251028 08:53:50.678298 6318 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:53:50.679147 6318 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:50.679214 6318 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:53:50.680401 6318 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:53:50.681207 6318 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Loaded TSK: 0
I20251028 08:53:50.681975 6318 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:53:50.682741 6318 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:53:50.683482 6318 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:53:50.714102 6244 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48492:
name: "default.Table_c896dce32fc04719beb2467b2bf1f855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:50.714201 6245 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48492:
table { table_name: "default.Table_d77cbfbaef6940f4bd623eaab2bb30aa" } new_table_name: "default.Table_a32163429a914f28ad8f6bac7f7c11c0" modify_external_catalogs: true
I20251028 08:53:50.714767 6240 master_service.cc:954] ReplaceTablet: received request to replace tablet c190d47c54dd4b9fa84a1241bd4cdbcf from {username='slave'} at 127.0.0.1:48492
I20251028 08:53:50.714811 6242 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48492:
table { table_name: "default.Table_64f349518d5d449ea6a147f1795a24fd" } new_table_name: "default.Table_018ba9cb543a4fd0a8e19856261027a1" modify_external_catalogs: true
I20251028 08:53:50.715184 6236 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b92cfd34a6c4e7fba8f785cc374217f from {username='slave'} at 127.0.0.1:48492
I20251028 08:53:50.715979 6236 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f7679e0e14447dbaf18854a5d41692a from {username='slave'} at 127.0.0.1:48492
I20251028 08:53:50.716369 6236 master_service.cc:954] ReplaceTablet: received request to replace tablet 896c194b66794478b4fa1d3a08128314 from {username='slave'} at 127.0.0.1:48492
I20251028 08:53:50.716777 6236 master_service.cc:954] ReplaceTablet: received request to replace tablet 896c194b66794478b4fa1d3a08128314 from {username='slave'} at 127.0.0.1:48492
I20251028 08:53:50.716707 6243 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48492:
table { table_name: "default.Table_976045e19bfe41c3b314c8cc12e020d1" } new_table_name: "default.Table_dfd48b4fd93d407bad513cecd15ecf68" modify_external_catalogs: true
I20251028 08:53:50.717253 6241 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48492:
name: "default.Table_6d9d93bde7fa47158f8e7b574783370e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:50.717295 6236 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48492:
name: "default.Table_541b7acce93a4f9785758f2d9acc4406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:50.717727 6238 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48492:
name: "default.Table_80a36344c9054a9e996f831074abfc41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:50.717901 6239 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48492:
name: "default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:50.718508 6237 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48492:
name: "default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:53:50.723659 6240 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Sending DeleteTablet for 3 replicas of tablet c190d47c54dd4b9fa84a1241bd4cdbcf
F20251028 08:53:50.724388 6300 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fba559abdcd google::LogMessage::Fail() at ??:0
@ 0x7fba559afb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fba559ab7cc google::LogMessage::Flush() at ??:0
@ 0x7fba559aba11 google::LogMessage::~LogMessage() at ??:0
I20251028 08:53:50.734944 6240 catalog_manager.cc:6719] ReplaceTablet: tablet c190d47c54dd4b9fa84a1241bd4cdbcf deleted and replaced by tablet f0b0ea645082479fa72ede160effe9ea
I20251028 08:53:50.740857 6240 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48492:
name: "default.Table_c22aee04efca4315947c85505204085f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fba6c36db64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fba6c36c411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251028 08:53:50.751191 5820 tablet_service.cc:1552] Processing DeleteTablet for tablet c190d47c54dd4b9fa84a1241bd4cdbcf with delete_type TABLET_DATA_DELETED (replaced by tablet f0b0ea645082479fa72ede160effe9ea) from {username='slave'} at 127.0.0.1:48206
I20251028 08:53:50.751678 6342 tablet_replica.cc:333] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:53:50.751870 6342 raft_consensus.cc:2243] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 1 LEADER]: Raft consensus shutting down.
I20251028 08:53:50.751861 5686 tablet_service.cc:1552] Processing DeleteTablet for tablet c190d47c54dd4b9fa84a1241bd4cdbcf with delete_type TABLET_DATA_DELETED (replaced by tablet f0b0ea645082479fa72ede160effe9ea) from {username='slave'} at 127.0.0.1:37002
I20251028 08:53:50.752170 5544 tablet_service.cc:1552] Processing DeleteTablet for tablet c190d47c54dd4b9fa84a1241bd4cdbcf with delete_type TABLET_DATA_DELETED (replaced by tablet f0b0ea645082479fa72ede160effe9ea) from {username='slave'} at 127.0.0.1:38308
I20251028 08:53:50.752359 6342 raft_consensus.cc:2272] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:50.755409 6342 ts_tablet_manager.cc:1916] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fba6c37408c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251028 08:53:50.760075 5820 tablet_service.cc:1505] Processing CreateTablet for tablet 774e1d1fe9334a45a7df75cc5cd098ff (DEFAULT_TABLE table=default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:50.760246 6343 tablet_replica.cc:333] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:53:50.760435 5820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774e1d1fe9334a45a7df75cc5cd098ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:50.760495 6343 raft_consensus.cc:2243] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:50.760730 6343 raft_consensus.cc:2272] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:50.761386 6342 ts_tablet_manager.cc:1929] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:53:50.761487 6342 log.cc:1199] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/c190d47c54dd4b9fa84a1241bd4cdbcf
I20251028 08:53:50.761906 6342 ts_tablet_manager.cc:1950] T c190d47c54dd4b9fa84a1241bd4cdbcf P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
I20251028 08:53:50.764770 6230 catalog_manager.cc:4985] TS bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): tablet c190d47c54dd4b9fa84a1241bd4cdbcf (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]) successfully deleted
I20251028 08:53:50.761409 6343 ts_tablet_manager.cc:1916] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:50.765137 5818 tablet_service.cc:1505] Processing CreateTablet for tablet 7a5654d71877406f8a5a4e1e9bb08fa4 (DEFAULT_TABLE table=default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:50.765549 5818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5654d71877406f8a5a4e1e9bb08fa4. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fba6c373f09 std::_Function_handler<>::_M_invoke() at ??:0
I20251028 08:53:50.767545 5817 tablet_service.cc:1505] Processing CreateTablet for tablet e591ce42e8ce4df0952cb231526544ab (DEFAULT_TABLE table=default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 08:53:50.767905 5817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e591ce42e8ce4df0952cb231526544ab. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0xa8867d std::function<>::operator()() at ??:?
I20251028 08:53:50.772032 5819 tablet_service.cc:1505] Processing CreateTablet for tablet f0b0ea645082479fa72ede160effe9ea (DEFAULT_TABLE table=default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7fba5871fee9 kudu::Thread::SuperviseThread() at ??:0
I20251028 08:53:50.772377 6343 ts_tablet_manager.cc:1929] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:53:50.772485 5819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b0ea645082479fa72ede160effe9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 08:53:50.772490 6343 log.cc:1199] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/c190d47c54dd4b9fa84a1241bd4cdbcf
I20251028 08:53:50.772909 6343 ts_tablet_manager.cc:1950] T c190d47c54dd4b9fa84a1241bd4cdbcf P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
@ 0x7fba5bd2a6db start_thread at ??:0
@ 0x7fba5485c71f clone at ??:0
I20251028 08:53:50.777622 6344 tablet_replica.cc:333] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
I20251028 08:53:50.777796 6344 raft_consensus.cc:2243] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 08:53:50.778019 6344 raft_consensus.cc:2272] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:53:50.778512 6344 ts_tablet_manager.cc:1916] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:53:50.782640 6347 tablet_bootstrap.cc:492] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:50.784456 6347 tablet_bootstrap.cc:654] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:50.786422 6344 ts_tablet_manager.cc:1929] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:53:50.786517 6344 log.cc:1199] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/c190d47c54dd4b9fa84a1241bd4cdbcf
I20251028 08:53:50.786947 6344 ts_tablet_manager.cc:1950] T c190d47c54dd4b9fa84a1241bd4cdbcf P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:53:50.793638 6347 tablet_bootstrap.cc:492] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:50.793773 6347 ts_tablet_manager.cc:1403] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20251028 08:53:50.794559 6347 raft_consensus.cc:359] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:50.794744 6347 raft_consensus.cc:385] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:50.794837 6347 raft_consensus.cc:740] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:50.795058 6347 consensus_queue.cc:260] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:50.795650 6347 ts_tablet_manager.cc:1434] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 08:53:50.797363 6347 tablet_bootstrap.cc:492] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:50.799101 6347 tablet_bootstrap.cc:654] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:50.803400 6347 tablet_bootstrap.cc:492] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:50.803531 6347 ts_tablet_manager.cc:1403] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20251028 08:53:50.804198 6347 raft_consensus.cc:359] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:50.804389 6347 raft_consensus.cc:385] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:50.804469 6347 raft_consensus.cc:740] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:50.804716 6347 consensus_queue.cc:260] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:50.805670 6347 ts_tablet_manager.cc:1434] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 08:53:50.805958 6347 tablet_bootstrap.cc:492] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:50.807399 6347 tablet_bootstrap.cc:654] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:50.812299 6347 tablet_bootstrap.cc:492] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:50.812395 6347 ts_tablet_manager.cc:1403] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20251028 08:53:50.812911 6347 raft_consensus.cc:359] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:50.813033 6347 raft_consensus.cc:385] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:50.813115 6347 raft_consensus.cc:740] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:50.813318 6347 consensus_queue.cc:260] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:50.813922 6347 ts_tablet_manager.cc:1434] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251028 08:53:50.814169 6347 tablet_bootstrap.cc:492] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9: Bootstrap starting.
I20251028 08:53:50.815582 6347 tablet_bootstrap.cc:654] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9: Neither blocks nor log segments found. Creating new log.
I20251028 08:53:50.817405 6347 tablet_bootstrap.cc:492] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9: No bootstrap required, opened a new log
I20251028 08:53:50.817523 6347 ts_tablet_manager.cc:1403] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251028 08:53:50.818097 6347 raft_consensus.cc:359] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:50.818248 6347 raft_consensus.cc:385] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 08:53:50.818341 6347 raft_consensus.cc:740] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbab3d08643e49e786c2174faaa741d9, State: Initialized, Role: FOLLOWER
I20251028 08:53:50.818544 6347 consensus_queue.cc:260] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:50.819069 6347 ts_tablet_manager.cc:1434] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251028 08:53:50.886613 5980 master_runner.cc:186] Error getting master registration for 127.4.217.62:41357: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111),
I20251028 08:53:50.886862 5980 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:53:50.950591 5641 connection.cc:537] server connection from 127.0.0.1:37002 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:50.950634 5502 connection.cc:537] server connection from 127.0.0.1:38308 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:50.950907 5773 connection.cc:537] server connection from 127.0.0.1:48206 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:50.951289 5775 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:50.951326 5895 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:53:50.951901 5895 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.217.61:33257)
W20251028 08:53:50.952482 5895 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.217.61:33257)
W20251028 08:53:50.952661 5895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.61:33257)
W20251028 08:53:50.953054 5895 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.217.61:33257)
W20251028 08:53:50.953224 6319 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 08:53:50.952040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.217.61:33257 (local address 127.4.217.3:55667)
1028 08:53:50.952929 (+ 889us) negotiation.cc:107] Waiting for socket to connect
1028 08:53:50.953004 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":782}
I20251028 08:53:50.984033 6309 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:50.984151 6309 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:50.984676 6309 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:50.985325 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:53:50.985513 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:53:50.985617 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:50.985833 6309 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:51.032224 6309 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:51.032339 6309 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:51.032773 6309 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:51.033401 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:53:51.033555 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:53:51.033663 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:51.033852 6309 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:51.237085 5616 heartbeater.cc:499] Master 127.4.217.61:33257 was elected leader, sending a full tablet report...
I20251028 08:53:51.254033 6363 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:53:51.254266 6363 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:51.254798 6363 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:51.256138 6044 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:51.256282 6045 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:51.256397 6045 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:51.256603 6363 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:51.461450 6309 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:51.461625 6309 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:51.462190 6309 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:51.462814 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:53:51.462998 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:53:51.463068 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:51.463270 6309 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:51.643493 4964 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1hpXWl/build/asan/bin/kudu pid 5980
I20251028 08:53:51.650158 6089 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 08:53:51.749548 6309 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:51.749761 6309 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:51.750381 6309 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:53:51.751035 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:53:51.751221 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:53:51.751319 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:51.751515 6309 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:51.754688 4964 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1hpXWl/build/asan/bin/kudu with pid 5980
I20251028 08:53:52.587208 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--webserver_interface=127.4.217.60
--webserver_port=41817
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251028 08:53:52.886931 6372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:52.887195 6372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:52.887322 6372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:52.894918 6372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:53:52.894999 6372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:52.895077 6372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:53:52.895135 6372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:53:52.903684 6372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:52.904970 6372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:52.906462 6372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:52.913501 6378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:52.913578 6377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:52.914646 6380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:52.914680 6372 server_base.cc:1047] running on GCE node
I20251028 08:53:52.915287 6372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:52.916414 6372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:52.917611 6372 hybrid_clock.cc:648] HybridClock initialized: now 1761641632917559 us; error 25 us; skew 500 ppm
I20251028 08:53:52.917924 6372 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:52.919763 6372 webserver.cc:492] Webserver started at http://127.4.217.60:41817/ using document root <none> and password file <none>
I20251028 08:53:52.920392 6372 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:52.920490 6372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:52.925159 6372 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20251028 08:53:52.927493 6386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:52.928560 6372 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251028 08:53:52.928704 6372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
uuid: "82b026d06b454d65b27216e3002f29af"
format_stamp: "Formatted at 2025-10-28 08:53:42 on dist-test-slave-cmml"
I20251028 08:53:52.929252 6372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:52.944839 6372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:52.945464 6372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:52.945752 6372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:52.964208 6372 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:37319
I20251028 08:53:52.964296 6437 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.60:37319 every 8 connection(s)
I20251028 08:53:52.965646 6372 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
I20251028 08:53:52.971700 6438 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:53:52.972128 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 6372
I20251028 08:53:52.975584 6438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap starting.
I20251028 08:53:52.986868 6438 log.cc:826] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:53.016332 6438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251028 08:53:53.017067 6438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap complete.
I20251028 08:53:53.024044 6438 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:53.026667 6438 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b026d06b454d65b27216e3002f29af, State: Initialized, Role: FOLLOWER
I20251028 08:53:53.027211 6438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:53.027957 6443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:53.028353 6443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:53.029228 6438 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:53:53.038796 6372 master_runner.cc:186] Error getting master registration for 127.4.217.62:41357: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111),
I20251028 08:53:53.044108 6372 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:53:53.044572 6457 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:53.044639 6457 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: loading cluster ID for follower catalog manager: success
I20251028 08:53:53.047232 6457 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: acquiring CA information for follower catalog manager: success
I20251028 08:53:53.048971 6457 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:53:53.263732 5614 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:53.265271 5751 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:53.266819 6402 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:58269; Asking this server to re-register.
I20251028 08:53:53.267330 6403 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:57265; Asking this server to re-register.
I20251028 08:53:53.267848 5751 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:53.267886 5614 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:53.269760 6403 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:53.269932 6402 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:53.332238 5885 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:53.333519 6402 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:38599; Asking this server to re-register.
I20251028 08:53:53.334419 5885 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:53.335774 6403 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:53.473670 6443 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:53.473881 6443 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:53.475476 6443 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:53.475906 6388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:53.477313 6388 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:53.477779 6389 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:53.477937 6389 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:53.478320 6443 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:54.150760 6466 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:54.150916 6466 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:54.151456 6466 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:54.152932 6388 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:54.153113 6389 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:54.153203 6389 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:54.153445 6466 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:54.280369 6465 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:54.280514 6465 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:54.281014 6465 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:54.281807 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:53:54.282042 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:53:54.282115 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:54.282289 6465 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:54.544159 6465 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:54.544327 6465 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:54.544857 6465 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:54.545565 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:53:54.545809 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:53:54.545907 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:54.546115 6465 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:54.694049 6471 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:54.694191 6471 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:54.694737 6471 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:54.696476 6388 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:54.696650 6389 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:54.696755 6389 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:54.696967 6471 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:54.768090 6465 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:54.768237 6465 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:54.768643 6465 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:54.769291 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:53:54.769546 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:53:54.769690 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:54.769918 6465 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:55.222333 4964 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1hpXWl/build/asan/bin/kudu pid 6372
I20251028 08:53:55.232594 6433 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 08:53:55.326627 4964 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1hpXWl/build/asan/bin/kudu with pid 6372
I20251028 08:53:55.331794 6465 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:55.331964 6465 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:55.332517 6465 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:53:55.333194 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:53:55.333379 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:53:55.333480 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:55.333724 6465 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:53:55.345065 5773 connection.cc:537] client connection to 127.4.217.60:37319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 08:53:55.990914 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--webserver_interface=127.4.217.60
--webserver_port=41817
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251028 08:53:56.287519 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:53:56.287791 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:53:56.287904 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:53:56.294847 6479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:53:56.294922 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:53:56.295001 6479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:53:56.295060 6479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:53:56.303359 6479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.60:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.217.60
--webserver_port=41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:53:56.304554 6479 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:53:56.306027 6479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:53:56.312748 6486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:56.312837 6485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:53:56.313197 6488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:53:56.313541 6479 server_base.cc:1047] running on GCE node
I20251028 08:53:56.314136 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:53:56.315209 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:53:56.316401 6479 hybrid_clock.cc:648] HybridClock initialized: now 1761641636316360 us; error 41 us; skew 500 ppm
I20251028 08:53:56.316762 6479 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:53:56.318486 6479 webserver.cc:492] Webserver started at http://127.4.217.60:41817/ using document root <none> and password file <none>
I20251028 08:53:56.319089 6479 fs_manager.cc:362] Metadata directory not provided
I20251028 08:53:56.319164 6479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:53:56.323729 6479 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 08:53:56.326012 6494 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:53:56.327068 6479 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251028 08:53:56.327194 6479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
uuid: "82b026d06b454d65b27216e3002f29af"
format_stamp: "Formatted at 2025-10-28 08:53:42 on dist-test-slave-cmml"
I20251028 08:53:56.327773 6479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:53:56.345362 6479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:53:56.346009 6479 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:53:56.346240 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:53:56.364179 6479 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.60:37319
I20251028 08:53:56.364228 6545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.60:37319 every 8 connection(s)
I20251028 08:53:56.365489 6479 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-2/data/info.pb
I20251028 08:53:56.371364 6546 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:53:56.374230 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap starting.
I20251028 08:53:56.375691 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 6479
I20251028 08:53:56.385960 6546 log.cc:826] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Log is configured to *not* fsync() on all Append() calls
I20251028 08:53:56.411844 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251028 08:53:56.412273 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Bootstrap complete.
I20251028 08:53:56.416858 6546 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:56.418879 6546 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b026d06b454d65b27216e3002f29af, State: Initialized, Role: FOLLOWER
I20251028 08:53:56.419378 6546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:56.420074 6551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:53:56.420329 6551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:53:56.421187 6546 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:53:56.436141 6565 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:53:56.436272 6565 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: loading cluster ID for follower catalog manager: success
I20251028 08:53:56.446848 6565 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: acquiring CA information for follower catalog manager: success
I20251028 08:53:56.449476 6565 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:53:56.446405 6479 master_runner.cc:186] Error getting master registration for 127.4.217.62:41357: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111),
I20251028 08:53:56.450019 6479 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:53:56.854905 5614 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:56.855494 5751 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:56.856418 5885 heartbeater.cc:344] Connected to a master server at 127.4.217.60:37319
I20251028 08:53:56.857926 6509 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:41583; Asking this server to re-register.
I20251028 08:53:56.857856 6510 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:40519; Asking this server to re-register.
I20251028 08:53:56.857887 6511 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:53629; Asking this server to re-register.
I20251028 08:53:56.858592 5614 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:56.858664 5751 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:56.859397 5885 heartbeater.cc:461] Registering TS with master...
I20251028 08:53:56.860407 6509 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:53:56.860545 6510 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:53:56.860639 6511 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:53:56.929706 6572 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:56.929917 6572 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:56.931489 6572 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:56.932081 6496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:56.933205 6496 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:56.933365 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:56.933501 6497 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:56.933871 6572 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:57.680047 6577 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:57.680186 6577 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:57.680742 6577 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:57.682994 6496 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:57.683131 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:57.683252 6497 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:57.683490 6577 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:58.654639 6582 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:58.654795 6582 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:58.655345 6582 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:58.657285 6496 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:53:58.657469 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:53:58.657555 6497 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:58.657793 6582 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:58.740871 6586 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:58.741070 6586 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:53:58.741937 6586 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:53:58.742774 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:53:58.742978 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:53:58.743078 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:58.743330 6586 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:59.425707 6587 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:59.425899 6587 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:53:59.426429 6587 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:53:59.428184 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:53:59.428337 6496 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:53:59.428486 6496 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:53:59.428719 6587 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251028 08:53:59.751329 6591 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:59.751468 6591 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:59.751945 6591 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:59.752636 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:53:59.752832 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:53:59.752919 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:59.753151 6591 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:53:59.780186 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--webserver_interface=127.4.217.61
--webserver_port=43565
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251028 08:53:59.853768 6591 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:53:59.853922 6591 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:53:59.854497 6591 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:53:59.855285 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:53:59.855494 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:53:59.855597 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:53:59.855813 6591 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:54:00.072752 6592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:54:00.073009 6592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:54:00.073143 6592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:54:00.081151 6592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:54:00.081221 6592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:54:00.081277 6592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:54:00.081337 6592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:54:00.090457 6592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:54:00.091930 6592 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:54:00.093662 6592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:54:00.101621 6598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:00.101687 6601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:00.101658 6599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:54:00.102592 6592 server_base.cc:1047] running on GCE node
I20251028 08:54:00.103292 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:54:00.104521 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:54:00.105696 6592 hybrid_clock.cc:648] HybridClock initialized: now 1761641640105666 us; error 23 us; skew 500 ppm
I20251028 08:54:00.105998 6592 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:54:00.108250 6592 webserver.cc:492] Webserver started at http://127.4.217.61:43565/ using document root <none> and password file <none>
I20251028 08:54:00.108892 6592 fs_manager.cc:362] Metadata directory not provided
I20251028 08:54:00.108989 6592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:54:00.113780 6592 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251028 08:54:00.116217 6607 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:54:00.117267 6592 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251028 08:54:00.117409 6592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:54:00.118017 6592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:54:00.134259 6592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:54:00.134878 6592 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:54:00.135095 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:54:00.151799 6592 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:33257
I20251028 08:54:00.151841 6658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.61:33257 every 8 connection(s)
I20251028 08:54:00.153087 6592 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
I20251028 08:54:00.154415 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 6592
I20251028 08:54:00.161049 6659 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:54:00.165071 6659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap starting.
I20251028 08:54:00.176258 6659 log.cc:826] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Log is configured to *not* fsync() on all Append() calls
I20251028 08:54:00.178982 5753 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:00.208401 6659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=39 ignored=0} inserts{seen=28 ignored=0} mutations{seen=27 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 08:54:00.208830 6659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap complete.
I20251028 08:54:00.213330 6659 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:00.213919 6659 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Initialized, Role: FOLLOWER
I20251028 08:54:00.214418 6659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:00.215018 6663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:00.215210 6663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:00.215806 6659 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:54:00.235545 6677 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:00.235679 6677 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: loading cluster ID for follower catalog manager: success
I20251028 08:54:00.233749 6592 master_runner.cc:186] Error getting master registration for 127.4.217.62:41357: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111),
I20251028 08:54:00.239354 6592 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:00.240298 6677 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: acquiring CA information for follower catalog manager: success
I20251028 08:54:00.243853 6677 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:54:00.259215 5616 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:00.259380 5616 heartbeater.cc:499] Master 127.4.217.61:33257 was elected leader, sending a full tablet report...
I20251028 08:54:00.261467 6624 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:51767; Asking this server to re-register.
I20251028 08:54:00.262032 5616 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:00.263397 6624 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:54:00.475895 5887 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:00.477437 6624 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:34803; Asking this server to re-register.
I20251028 08:54:00.478112 5887 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:00.479375 6624 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:54:00.635919 6682 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:00.636055 6682 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:00.636648 6682 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:00.637276 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:00.637463 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:00.637539 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:00.637759 6682 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:00.682286 6624 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:41805; Asking this server to re-register.
I20251028 08:54:00.682775 5753 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:00.683748 6624 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:54:00.814725 6683 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:00.814942 6683 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:00.816514 6683 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:00.817265 6609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:54:00.818809 6609 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:54:00.823532 6521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af" is_pre_election: true
I20251028 08:54:00.823822 6521 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 4.
I20251028 08:54:00.824329 6608 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 0b9421be172b482f9c69d4dc5d2b19a8
I20251028 08:54:00.824625 6683 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251028 08:54:00.824744 6683 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:54:00.824846 6683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 4 FOLLOWER]: Advancing to term 5
I20251028 08:54:00.827939 6683 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:00.828377 6683 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 5 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:00.828809 6521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "82b026d06b454d65b27216e3002f29af"
I20251028 08:54:00.828986 6521 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 4 FOLLOWER]: Advancing to term 5
W20251028 08:54:00.829357 6609 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:54:00.831964 6521 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 5.
I20251028 08:54:00.832258 6608 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 0b9421be172b482f9c69d4dc5d2b19a8
I20251028 08:54:00.832541 6683 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Leader election won for term 5
I20251028 08:54:00.832932 6683 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 LEADER]: Becoming Leader. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Running, Role: LEADER
I20251028 08:54:00.833317 6683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:00.835318 6688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 5 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:00.835489 6688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:54:00.836098 6690 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:54:00.838406 6690 catalog_manager.cc:679] Loaded metadata for table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:54:00.838761 6690 catalog_manager.cc:679] Loaded metadata for table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]
I20251028 08:54:00.839098 6690 catalog_manager.cc:679] Loaded metadata for table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:54:00.839470 6690 catalog_manager.cc:679] Loaded metadata for table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:54:00.839820 6690 catalog_manager.cc:679] Loaded metadata for table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:54:00.840152 6690 catalog_manager.cc:679] Loaded metadata for table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:54:00.840477 6690 catalog_manager.cc:679] Loaded metadata for table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:54:00.840785 6690 catalog_manager.cc:679] Loaded metadata for table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:54:00.842139 6690 tablet_loader.cc:96] loaded metadata for tablet 2b26ada1ba9546e880d888184f745c40 (table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14])
I20251028 08:54:00.842944 6690 tablet_loader.cc:96] loaded metadata for tablet 6203a2cbeb114d37b8e4d018428c3b0d (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4])
I20251028 08:54:00.843333 6690 tablet_loader.cc:96] loaded metadata for tablet 774e1d1fe9334a45a7df75cc5cd098ff (table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2])
I20251028 08:54:00.843600 6690 tablet_loader.cc:96] loaded metadata for tablet 7a5654d71877406f8a5a4e1e9bb08fa4 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad])
I20251028 08:54:00.843859 6690 tablet_loader.cc:96] loaded metadata for tablet 7ff55e6d288f44428891d6fc7e93e2d2 (table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196])
I20251028 08:54:00.844576 6690 tablet_loader.cc:96] loaded metadata for tablet da51f1509b654b398d16fe14777dbbdd (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508])
I20251028 08:54:00.844810 6690 tablet_loader.cc:96] loaded metadata for tablet e591ce42e8ce4df0952cb231526544ab (table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3])
I20251028 08:54:00.845067 6690 tablet_loader.cc:96] loaded metadata for tablet f0b0ea645082479fa72ede160effe9ea (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e])
I20251028 08:54:00.845328 6690 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:54:00.846203 6690 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:00.846279 6690 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:54:00.847352 6690 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:54:00.848192 6690 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Loaded TSK: 0
I20251028 08:54:00.848857 6690 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:54:00.849740 6690 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:54:00.850582 6690 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:54:00.949651 6624 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b92cfd34a6c4e7fba8f785cc374217f from {username='slave'} at 127.0.0.1:60920
I20251028 08:54:00.949831 6622 master_service.cc:954] ReplaceTablet: received request to replace tablet 7ff55e6d288f44428891d6fc7e93e2d2 from {username='slave'} at 127.0.0.1:60920
I20251028 08:54:00.950248 6620 master_service.cc:954] ReplaceTablet: received request to replace tablet da51f1509b654b398d16fe14777dbbdd from {username='slave'} at 127.0.0.1:60920
I20251028 08:54:00.951239 6621 master_service.cc:954] ReplaceTablet: received request to replace tablet 7ff55e6d288f44428891d6fc7e93e2d2 from {username='slave'} at 127.0.0.1:60920
I20251028 08:54:00.950997 6617 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60920:
name: "default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:00.949679 6623 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f7679e0e14447dbaf18854a5d41692a from {username='slave'} at 127.0.0.1:60920
I20251028 08:54:00.952281 6623 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60920:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:54:00.952536 6623 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60920:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:54:00.952875 6619 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60920:
table { table_name: "default.Table_dd23f4b28a514540926eb05e3f12c357" } new_table_name: "default.Table_72938f3a321a4a74ba29b7aace4a1998" modify_external_catalogs: true
I20251028 08:54:00.953482 6618 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60920:
name: "default.Table_b01e4320f600411a872c6485f421dd5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:00.954214 6616 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60920:
name: "default.Table_c496c6ebeb1d44248b341428961d8939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:00.954495 6615 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60920:
table { table_name: "default.Table_a90e8e5ec3364d6b80abc305644d234c" } modify_external_catalogs: true
I20251028 08:54:00.954735 6615 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60920:
table { table_name: "default.Table_a90e8e5ec3364d6b80abc305644d234c" } modify_external_catalogs: true
I20251028 08:54:00.955168 6624 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60920:
table { table_name: "default.Table_f617dc4ea4f444fa8528dda685e72fa8" } new_table_name: "default.Table_8588dfa71ef54b07996e4bd0a2b29cfc" modify_external_catalogs: true
I20251028 08:54:00.958775 6521 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Refusing update from remote peer 8b7d4120e0c84998858b0bd4ed9572b7: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
F20251028 08:54:00.959148 6676 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251028 08:54:00.961464 6688 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 38, Time since last communication: 0.000s
@ 0x7f74c2ecbdcd google::LogMessage::Fail() at ??:0
@ 0x7f74c2ecfb93 google::LogMessage::SendToLog() at ??:0
W20251028 08:54:00.963586 6609 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 -> Peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Couldn't send request to peer 0b9421be172b482f9c69d4dc5d2b19a8. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f74c2ecb7cc google::LogMessage::Flush() at ??:0
@ 0x7f74c2ecba11 google::LogMessage::~LogMessage() at ??:0
I20251028 08:54:00.972244 6695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 5 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:00.972532 6695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:00.973378 6691 mvcc.cc:204] Tried to move back new op lower bound from 7215684161355751424 to 7215684160856207360. Current Snapshot: MvccSnapshot[applied={T|T < 7215684161355751424}]
I20251028 08:54:00.975293 6697 mvcc.cc:204] Tried to move back new op lower bound from 7215684161359065088 to 7215684160856207360. Current Snapshot: MvccSnapshot[applied={T|T < 7215684161359065088 or (T in {7215684161359065088})}]
@ 0x7f74d988db64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251028 08:54:00.978277 6695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:00.978446 6695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:00.980340 6622 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Sending DeleteTablet for 3 replicas of tablet 7ff55e6d288f44428891d6fc7e93e2d2
@ 0x7f74d988c411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251028 08:54:00.982486 6622 catalog_manager.cc:6719] ReplaceTablet: tablet 7ff55e6d288f44428891d6fc7e93e2d2 deleted and replaced by tablet 636688cf451f4e3a97400a1055d1b47b
@ 0x7f74d989408c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f74d9893f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f74c5c3fee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f74c924a6db start_thread at ??:0
@ 0x7f74c1d7c71f clone at ??:0
I20251028 08:54:01.005219 5818 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ff55e6d288f44428891d6fc7e93e2d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 636688cf451f4e3a97400a1055d1b47b) from {username='slave'} at 127.0.0.1:42988
I20251028 08:54:01.018363 6710 tablet_replica.cc:333] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:54:01.018585 6710 raft_consensus.cc:2243] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 08:54:01.018805 6710 raft_consensus.cc:2272] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:54:01.019465 6710 ts_tablet_manager.cc:1916] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:54:01.026749 6710 ts_tablet_manager.cc:1929] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:54:01.026866 6710 log.cc:1199] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/7ff55e6d288f44428891d6fc7e93e2d2
I20251028 08:54:01.027305 6710 ts_tablet_manager.cc:1950] T 7ff55e6d288f44428891d6fc7e93e2d2 P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
W20251028 08:54:01.150995 5775 connection.cc:537] server connection from 127.0.0.1:42988 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:01.151525 6707 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 08:54:00.985739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.4.217.2:35819)
1028 08:54:01.003430 (+ 17691us) server_negotiation.cc:207] Beginning negotiation
1028 08:54:01.003435 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1028 08:54:01.003451 (+ 16us) server_negotiation.cc:408] Connection header received
1028 08:54:01.003505 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1028 08:54:01.003516 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1028 08:54:01.003580 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1028 08:54:01.003688 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1028 08:54:01.151132 (+147444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33982: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":17565,"thread_start_us":17465,"threads_started":1}
W20251028 08:54:01.152652 5895 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:01.152911 5895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:01.153368 5895 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:01.153625 5895 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.217.61:33257)
I20251028 08:54:01.451148 6479 master_runner.cc:186] Error getting master registration for 127.4.217.62:41357: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111),
I20251028 08:54:01.451304 6479 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:01.474365 6695 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:01.474493 6695 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:01.475209 6695 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:54:01.476603 6496 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:54:01.476742 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:54:01.476898 6497 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:01.477108 6695 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251028 08:54:01.673545 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 1: this message will repeat every 5th retry.
I20251028 08:54:02.220906 6728 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:02.221071 6728 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:02.221630 6728 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:54:02.222412 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:02.222412 6496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251028 08:54:02.226600 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:02.226768 6496 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
I20251028 08:54:02.226857 6496 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:02.227044 6728 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251028 08:54:02.862181 6737 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:02.862365 6737 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:54:02.863185 6737 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:54:02.863874 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:54:02.864058 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:54:02.864173 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:02.864390 6737 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:02.969960 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41357
--webserver_interface=127.4.217.62
--webserver_port=46365
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251028 08:54:03.119362 6741 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:03.119561 6741 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:03.120254 6741 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:54:03.122705 6496 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357): Network error: Client connection negotiation failed: client connection to 127.4.217.62:41357: connect: Connection refused (error 111)
W20251028 08:54:03.122956 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:54:03.123072 6497 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82b026d06b454d65b27216e3002f29af; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:03.123291 6741 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251028 08:54:03.282536 6738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:54:03.282814 6738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:54:03.282936 6738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:54:03.290031 6738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:54:03.290114 6738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:54:03.290199 6738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:54:03.290264 6738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:54:03.298915 6738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.62:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.217.62
--webserver_port=46365
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:54:03.300266 6738 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:54:03.301803 6738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:54:03.308956 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:03.308923 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:03.309166 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:54:03.310021 6738 server_base.cc:1047] running on GCE node
I20251028 08:54:03.310699 6738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:54:03.311937 6738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:54:03.313133 6738 hybrid_clock.cc:648] HybridClock initialized: now 1761641643313076 us; error 47 us; skew 500 ppm
I20251028 08:54:03.313428 6738 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:54:03.315321 6738 webserver.cc:492] Webserver started at http://127.4.217.62:46365/ using document root <none> and password file <none>
I20251028 08:54:03.315932 6738 fs_manager.cc:362] Metadata directory not provided
I20251028 08:54:03.316042 6738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:54:03.320742 6738 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20251028 08:54:03.323410 6758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:54:03.324816 6738 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251028 08:54:03.324950 6738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:54:03.325641 6738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:54:03.344609 6738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:54:03.345369 6738 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:54:03.345669 6738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:54:03.364022 6738 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.62:41357
I20251028 08:54:03.364042 6810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.62:41357 every 8 connection(s)
I20251028 08:54:03.365499 6738 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-0/data/info.pb
I20251028 08:54:03.366139 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 6738
I20251028 08:54:03.373741 6811 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:54:03.377707 6811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Bootstrap starting.
I20251028 08:54:03.388139 6811 log.cc:826] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Log is configured to *not* fsync() on all Append() calls
I20251028 08:54:03.415107 6811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 08:54:03.415531 6811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: Bootstrap complete.
I20251028 08:54:03.420285 6811 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:03.420909 6811 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9421be172b482f9c69d4dc5d2b19a8, State: Initialized, Role: FOLLOWER
I20251028 08:54:03.421368 6811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:03.422086 6815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:03.422360 6815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:03.423041 6811 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:54:03.433773 6829 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:03.433904 6829 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: loading cluster ID for follower catalog manager: success
I20251028 08:54:03.437851 6829 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: acquiring CA information for follower catalog manager: success
I20251028 08:54:03.440470 6829 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:54:03.453065 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:03.457245 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:03.484933 5752 heartbeater.cc:344] Connected to a master server at 127.4.217.62:41357
I20251028 08:54:03.488579 6776 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:40619; Asking this server to re-register.
I20251028 08:54:03.489286 5752 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:03.490936 6776 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:54:03.750751 5615 heartbeater.cc:344] Connected to a master server at 127.4.217.62:41357
I20251028 08:54:03.752338 6776 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:50335; Asking this server to re-register.
I20251028 08:54:03.752908 5615 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:03.754091 6776 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:54:03.760805 6837 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:03.760959 6837 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:03.761515 6837 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
W20251028 08:54:03.774487 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:54:03.776132 6786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b026d06b454d65b27216e3002f29af" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" is_pre_election: true
I20251028 08:54:03.776718 6786 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b026d06b454d65b27216e3002f29af in term 2.
I20251028 08:54:03.777339 6496 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 82b026d06b454d65b27216e3002f29af; no voters: 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:03.777644 6837 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251028 08:54:03.777772 6837 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:03.777997 6837 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 5 FOLLOWER]: Advancing to term 6
I20251028 08:54:03.780064 6837 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:03.780637 6837 leader_election.cc:290] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257)
I20251028 08:54:03.780988 6786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b026d06b454d65b27216e3002f29af" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
I20251028 08:54:03.781185 6786 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 2 FOLLOWER]: Advancing to term 6
W20251028 08:54:03.781914 6497 leader_election.cc:336] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:54:03.785018 6786 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b026d06b454d65b27216e3002f29af in term 6.
I20251028 08:54:03.785398 6496 leader_election.cc:304] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 82b026d06b454d65b27216e3002f29af; no voters: 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:03.785660 6837 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 6 FOLLOWER]: Leader election won for term 6
I20251028 08:54:03.786047 6837 raft_consensus.cc:697] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [term 6 LEADER]: Becoming Leader. State: Replica: 82b026d06b454d65b27216e3002f29af, State: Running, Role: LEADER
I20251028 08:54:03.786419 6837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:03.787904 6841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b026d06b454d65b27216e3002f29af. Latest consensus state: current_term: 6 leader_uuid: "82b026d06b454d65b27216e3002f29af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:03.788242 6841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af [sys.catalog]: This master's current role is: LEADER
I20251028 08:54:03.788831 6843 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:54:03.791294 6843 catalog_manager.cc:679] Loaded metadata for table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:54:03.791708 6843 catalog_manager.cc:679] Loaded metadata for table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]
I20251028 08:54:03.792049 6843 catalog_manager.cc:679] Loaded metadata for table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:54:03.792376 6843 catalog_manager.cc:679] Loaded metadata for table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:54:03.792681 6843 catalog_manager.cc:679] Loaded metadata for table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:54:03.793007 6843 catalog_manager.cc:679] Loaded metadata for table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:54:03.793321 6843 catalog_manager.cc:679] Loaded metadata for table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:54:03.793674 6843 catalog_manager.cc:679] Loaded metadata for table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:54:03.795217 6843 tablet_loader.cc:96] loaded metadata for tablet 2b26ada1ba9546e880d888184f745c40 (table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14])
I20251028 08:54:03.796190 6843 tablet_loader.cc:96] loaded metadata for tablet 6203a2cbeb114d37b8e4d018428c3b0d (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4])
I20251028 08:54:03.796703 6843 tablet_loader.cc:96] loaded metadata for tablet 636688cf451f4e3a97400a1055d1b47b (table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196])
I20251028 08:54:03.796949 6843 tablet_loader.cc:96] loaded metadata for tablet 66fce75a826e48a8b141cf108cb94798 (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508])
I20251028 08:54:03.797224 6843 tablet_loader.cc:96] loaded metadata for tablet 774e1d1fe9334a45a7df75cc5cd098ff (table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2])
I20251028 08:54:03.797503 6843 tablet_loader.cc:96] loaded metadata for tablet 7a5654d71877406f8a5a4e1e9bb08fa4 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad])
I20251028 08:54:03.798678 6843 tablet_loader.cc:96] loaded metadata for tablet e591ce42e8ce4df0952cb231526544ab (table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3])
I20251028 08:54:03.799008 6843 tablet_loader.cc:96] loaded metadata for tablet f0b0ea645082479fa72ede160effe9ea (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e])
I20251028 08:54:03.799232 6843 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:54:03.800145 6843 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:03.800216 6843 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:54:03.801410 6843 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:54:03.802325 6843 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 82b026d06b454d65b27216e3002f29af: Loaded TSK: 0
I20251028 08:54:03.803102 6843 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:54:03.804319 6843 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:54:03.805119 6843 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:54:03.817967 6510 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47242:
table { table_name: "default.Table_78fd665055cb412390a37c5a185dec85" } modify_external_catalogs: true
I20251028 08:54:03.818248 6510 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47242:
table { table_name: "default.Table_78fd665055cb412390a37c5a185dec85" } modify_external_catalogs: true
I20251028 08:54:03.818297 6511 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47242:
name: "default.Table_c22aee04efca4315947c85505204085f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:03.818747 6509 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47242:
name: "default.Table_541b7acce93a4f9785758f2d9acc4406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:03.819041 6508 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47242:
name: "default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:03.819634 6503 master_service.cc:954] ReplaceTablet: received request to replace tablet da51f1509b654b398d16fe14777dbbdd from {username='slave'} at 127.0.0.1:47242
I20251028 08:54:03.819867 6504 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47242:
table { table_name: "default.Table_dd23f4b28a514540926eb05e3f12c357" } new_table_name: "default.Table_72938f3a321a4a74ba29b7aace4a1998" modify_external_catalogs: true
I20251028 08:54:03.820792 6505 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47242:
table { table_name: "default.Table_d77cbfbaef6940f4bd623eaab2bb30aa" } new_table_name: "default.Table_a32163429a914f28ad8f6bac7f7c11c0" modify_external_catalogs: true
I20251028 08:54:03.820745 6507 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47242:
name: "default.Table_80a36344c9054a9e996f831074abfc41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:03.821215 6502 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47242:
name: "default.Table_b01e4320f600411a872c6485f421dd5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:03.821899 6506 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47242:
name: "default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:03.822997 6503 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47242:
name: "default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251028 08:54:03.825205 6564 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f0418f0ddcd google::LogMessage::Fail() at ??:0
@ 0x7f0418f11b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0418f0d7cc google::LogMessage::Flush() at ??:0
@ 0x7f0418f0da11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f042f8cfb64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f042f8ce411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f042f8d608c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f042f8d5f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251028 08:54:03.851354 5886 heartbeater.cc:344] Connected to a master server at 127.4.217.62:41357
I20251028 08:54:03.854249 6776 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:47165; Asking this server to re-register.
@ 0x7f041bc81ee9 kudu::Thread::SuperviseThread() at ??:0
I20251028 08:54:03.855125 5886 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:03.856750 6776 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
@ 0x7f041f28c6db start_thread at ??:0
@ 0x7f0417dbe71f clone at ??:0
W20251028 08:54:04.006999 5773 connection.cc:537] client connection to 127.4.217.60:37319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:04.007993 5501 connection.cc:537] client connection to 127.4.217.60:37319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:04.008529 5893 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.217.60:37319)
W20251028 08:54:04.009053 5893 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.60:37319)
W20251028 08:54:04.009230 5893 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.217.60:37319)
W20251028 08:54:04.009853 5893 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.217.60:37319)
I20251028 08:54:04.455103 6857 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:04.455258 6857 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:04.455899 6857 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:04.456558 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:04.456738 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:04.456833 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:04.457041 6857 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:04.489956 6860 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:04.490126 6860 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:04.491750 6860 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:04.492630 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:04.493700 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:54:04.493868 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:54:04.493999 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:04.494269 6860 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251028 08:54:05.228039 6869 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:05.228179 6869 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:05.228674 6869 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:05.230456 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:05.230650 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:05.230803 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:05.231007 6869 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251028 08:54:05.950285 6877 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:05.950448 6877 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:05.951086 6877 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:05.952847 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:05.952973 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:05.953125 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:05.953349 6877 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251028 08:54:05.983197 6873 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:05.983326 6873 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:05.983807 6873 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:05.984519 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:54:05.984746 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:54:05.984834 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:05.985116 6873 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:06.076505 6873 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:06.076673 6873 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:06.077225 6873 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:06.077996 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:54:06.078198 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:54:06.078296 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:06.078548 6873 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:06.225840 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--webserver_interface=127.4.217.61
--webserver_port=43565
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251028 08:54:06.533454 6881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:54:06.533771 6881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:54:06.533910 6881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:54:06.541677 6881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:54:06.541744 6881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:54:06.541826 6881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:54:06.541888 6881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:54:06.550972 6881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:54:06.552335 6881 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:54:06.553987 6881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:54:06.561800 6891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:06.561888 6890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:06.562275 6893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:54:06.563892 6881 server_base.cc:1047] running on GCE node
I20251028 08:54:06.564608 6881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:54:06.565860 6881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:54:06.567031 6881 hybrid_clock.cc:648] HybridClock initialized: now 1761641646566993 us; error 35 us; skew 500 ppm
I20251028 08:54:06.567349 6881 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:54:06.569228 6881 webserver.cc:492] Webserver started at http://127.4.217.61:43565/ using document root <none> and password file <none>
I20251028 08:54:06.569937 6881 fs_manager.cc:362] Metadata directory not provided
I20251028 08:54:06.570029 6881 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:54:06.574716 6881 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 08:54:06.577319 6899 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:54:06.578724 6881 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251028 08:54:06.578850 6881 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:54:06.579433 6881 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:54:06.600195 6881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:54:06.600857 6881 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:54:06.601065 6881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:54:06.618724 6881 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:33257
I20251028 08:54:06.618754 6950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.61:33257 every 8 connection(s)
I20251028 08:54:06.620352 6881 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
I20251028 08:54:06.621441 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 6881
I20251028 08:54:06.629850 6951 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:54:06.635143 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap starting.
I20251028 08:54:06.646656 6951 log.cc:826] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Log is configured to *not* fsync() on all Append() calls
I20251028 08:54:06.666114 5887 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:06.679461 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=41 ignored=0} inserts{seen=29 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251028 08:54:06.679970 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap complete.
I20251028 08:54:06.684592 6951 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:06.686677 6951 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Initialized, Role: FOLLOWER
I20251028 08:54:06.687197 6951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 5.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:06.687851 6956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:06.688053 6956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:06.688658 6951 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:54:06.705569 6970 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:06.705761 6970 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: loading cluster ID for follower catalog manager: success
I20251028 08:54:06.711705 6970 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: acquiring CA information for follower catalog manager: success
I20251028 08:54:06.715253 6970 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:54:06.737630 6975 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:06.737846 6975 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:06.746961 6975 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:06.762116 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:06.764283 6926 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" candidate_term: 7 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" is_pre_election: true
I20251028 08:54:06.765046 6926 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b9421be172b482f9c69d4dc5d2b19a8 for term 7 because replica has last-logged OpId of term: 5 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 32.
I20251028 08:54:06.766198 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:06.766551 6975 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251028 08:54:06.769644 6881 master_runner.cc:186] Error getting master registration for 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111),
I20251028 08:54:06.775043 6881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:06.783651 5616 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:06.785246 6916 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:60507; Asking this server to re-register.
I20251028 08:54:06.785782 5616 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:06.787075 6916 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:54:06.984014 5753 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:06.985455 6916 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:49937; Asking this server to re-register.
I20251028 08:54:06.986028 5753 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:06.987063 6916 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:54:07.090590 6956 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:07.090795 6956 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:07.092224 6956 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:07.092590 6786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" is_pre_election: true
W20251028 08:54:07.092772 6900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:07.092868 6786 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b7d4120e0c84998858b0bd4ed9572b7 in current term 6: Already voted for candidate 82b026d06b454d65b27216e3002f29af in this term.
W20251028 08:54:07.094000 6900 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:07.094123 6900 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 0b9421be172b482f9c69d4dc5d2b19a8, 82b026d06b454d65b27216e3002f29af
I20251028 08:54:07.094412 6956 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 5 FOLLOWER]: Advancing to term 6
I20251028 08:54:07.097347 6956 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251028 08:54:07.169497 6916 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:47705; Asking this server to re-register.
I20251028 08:54:07.170116 5887 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:07.171154 6916 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:54:07.846657 6987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:07.846802 6987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:07.847498 6987 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:07.847908 6786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" is_pre_election: true
I20251028 08:54:07.848150 6786 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 6.
I20251028 08:54:07.848574 6901 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters:
I20251028 08:54:07.848879 6987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251028 08:54:07.849014 6987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:54:07.849100 6987 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 6 FOLLOWER]: Advancing to term 7
W20251028 08:54:07.849390 6900 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:07.851081 6987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:07.851567 6987 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 7 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:07.851883 6786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
I20251028 08:54:07.852039 6786 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 6 FOLLOWER]: Advancing to term 7
W20251028 08:54:07.852633 6900 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:07.853928 6786 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 7.
I20251028 08:54:07.854247 6901 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 82b026d06b454d65b27216e3002f29af
I20251028 08:54:07.854553 6987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Leader election won for term 7
I20251028 08:54:07.854869 6987 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 LEADER]: Becoming Leader. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Running, Role: LEADER
I20251028 08:54:07.855247 6987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 5.42, Last appended by leader: 42, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:07.857576 6990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 7 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:07.857789 6990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:54:08.255187 6786 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Refusing update from remote peer 8b7d4120e0c84998858b0bd4ed9572b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 7 index: 43. (index mismatch)
I20251028 08:54:08.255793 6990 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 32, Time since last communication: 0.000s
I20251028 08:54:08.266060 7000 mvcc.cc:204] Tried to move back new op lower bound from 7215684161355751424 to 7215684099728531456. Current Snapshot: MvccSnapshot[applied={T|T < 7215684119503167488}]
I20251028 08:54:08.266355 7008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:08.267248 7002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:08.268051 7002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:08.268455 7002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:08.268620 7002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:08.268942 6987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:08.269191 6987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:54:08.267690 7008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:08.270215 7006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:08.270469 7006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
W20251028 08:54:08.270539 6900 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 -> Peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Couldn't send request to peer 82b026d06b454d65b27216e3002f29af. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251028 08:54:08.270737 6999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 7 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:08.270964 6999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:08.271194 6992 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:54:08.273391 6992 catalog_manager.cc:679] Loaded metadata for table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:54:08.273831 6992 catalog_manager.cc:679] Loaded metadata for table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]
I20251028 08:54:08.274148 6992 catalog_manager.cc:679] Loaded metadata for table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:54:08.274447 6992 catalog_manager.cc:679] Loaded metadata for table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:54:08.274740 6992 catalog_manager.cc:679] Loaded metadata for table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:54:08.275043 6992 catalog_manager.cc:679] Loaded metadata for table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:54:08.275332 6992 catalog_manager.cc:679] Loaded metadata for table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:54:08.275624 6992 catalog_manager.cc:679] Loaded metadata for table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:54:08.277029 6992 tablet_loader.cc:96] loaded metadata for tablet 2b26ada1ba9546e880d888184f745c40 (table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14])
I20251028 08:54:08.277935 6992 tablet_loader.cc:96] loaded metadata for tablet 6203a2cbeb114d37b8e4d018428c3b0d (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4])
I20251028 08:54:08.278294 6992 tablet_loader.cc:96] loaded metadata for tablet 636688cf451f4e3a97400a1055d1b47b (table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196])
I20251028 08:54:08.278573 6992 tablet_loader.cc:96] loaded metadata for tablet 66fce75a826e48a8b141cf108cb94798 (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508])
I20251028 08:54:08.278873 6992 tablet_loader.cc:96] loaded metadata for tablet 774e1d1fe9334a45a7df75cc5cd098ff (table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2])
I20251028 08:54:08.279160 6992 tablet_loader.cc:96] loaded metadata for tablet 7a5654d71877406f8a5a4e1e9bb08fa4 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad])
I20251028 08:54:08.280288 6992 tablet_loader.cc:96] loaded metadata for tablet e591ce42e8ce4df0952cb231526544ab (table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3])
I20251028 08:54:08.280584 6992 tablet_loader.cc:96] loaded metadata for tablet f0b0ea645082479fa72ede160effe9ea (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e])
I20251028 08:54:08.280784 6992 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:54:08.281531 6992 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:08.281615 6992 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:54:08.282707 6992 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:54:08.283506 6992 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Loaded TSK: 0
I20251028 08:54:08.284346 6992 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:54:08.285238 6992 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:54:08.286098 6992 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:54:08.341063 6916 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b92cfd34a6c4e7fba8f785cc374217f from {username='slave'} at 127.0.0.1:34184
I20251028 08:54:08.341109 6915 master_service.cc:954] ReplaceTablet: received request to replace tablet 896c194b66794478b4fa1d3a08128314 from {username='slave'} at 127.0.0.1:34184
I20251028 08:54:08.341485 6914 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b92cfd34a6c4e7fba8f785cc374217f from {username='slave'} at 127.0.0.1:34184
I20251028 08:54:08.341866 6913 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b92cfd34a6c4e7fba8f785cc374217f from {username='slave'} at 127.0.0.1:34184
I20251028 08:54:08.342799 6915 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "default.Table_78fd665055cb412390a37c5a185dec85" } modify_external_catalogs: true
I20251028 08:54:08.342772 6916 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "default.Table_541b7acce93a4f9785758f2d9acc4406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:08.343022 6915 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "default.Table_78fd665055cb412390a37c5a185dec85" } modify_external_catalogs: true
I20251028 08:54:08.342972 6914 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "default.Table_c22aee04efca4315947c85505204085f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:08.344012 6913 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "default.Table_d77cbfbaef6940f4bd623eaab2bb30aa" } new_table_name: "default.Table_a32163429a914f28ad8f6bac7f7c11c0" modify_external_catalogs: true
I20251028 08:54:08.343974 6908 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "default.Table_80a36344c9054a9e996f831074abfc41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:08.344749 6910 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "default.Table_64f349518d5d449ea6a147f1795a24fd" } new_table_name: "default.Table_018ba9cb543a4fd0a8e19856261027a1" modify_external_catalogs: true
I20251028 08:54:08.344812 6912 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "default.Table_c896dce32fc04719beb2467b2bf1f855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:08.344898 6909 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:08.345435 6911 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "default.Table_976045e19bfe41c3b314c8cc12e020d1" } new_table_name: "default.Table_dfd48b4fd93d407bad513cecd15ecf68" modify_external_catalogs: true
I20251028 08:54:08.345494 6907 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251028 08:54:08.349057 6969 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fca421dddcd google::LogMessage::Fail() at ??:0
@ 0x7fca421e1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fca421dd7cc google::LogMessage::Flush() at ??:0
@ 0x7fca421dda11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fca58b9fb64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fca58b9e411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fca58ba608c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fca58ba5f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7fca44f51ee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fca4855c6db start_thread at ??:0
@ 0x7fca4108e71f clone at ??:0
I20251028 08:54:08.524869 6998 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:08.525012 6998 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:08.525635 6998 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:08.526436 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:08.526741 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:08.526858 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:08.527065 6998 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:54:08.527868 6761 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 08:54:08.528088 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20251028 08:54:08.528229 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:54:08.528550 5895 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.217.61:33257)
W20251028 08:54:08.528807 5895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:08.528966 5895 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:08.529194 5895 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.217.61:33257)
I20251028 08:54:08.769038 6999 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:08.769188 6999 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:08.769873 6999 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:08.771524 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:08.771981 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:08.772104 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:08.772320 6999 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251028 08:54:08.792826 5616 heartbeater.cc:499] Master 127.4.217.61:33257 was elected leader, sending a full tablet report...
I20251028 08:54:09.339461 7033 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:09.339648 7033 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:09.340308 7033 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:09.341958 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:09.342133 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:09.342231 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:09.342435 7033 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251028 08:54:09.651386 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--webserver_interface=127.4.217.61
--webserver_port=43565
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251028 08:54:09.839608 7040 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:09.839776 7040 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:09.840468 7040 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:09.841428 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:54:09.841693 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:54:09.841826 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:09.842087 7040 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:09.861670 7041 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:09.861814 7041 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:09.862366 7041 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:09.863101 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20251028 08:54:09.864602 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:09.864763 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:09.864867 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:09.865115 7041 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251028 08:54:09.962194 7037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:54:09.962447 7037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:54:09.962546 7037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:54:09.970497 7037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:54:09.970587 7037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:54:09.970693 7037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:54:09.970762 7037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:54:09.979678 7037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:54:09.981040 7037 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:54:09.982704 7037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:54:09.990077 7050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:09.990108 7049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:09.990353 7052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:54:09.991379 7037 server_base.cc:1047] running on GCE node
I20251028 08:54:09.991972 7037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:54:09.993088 7037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:54:09.994256 7037 hybrid_clock.cc:648] HybridClock initialized: now 1761641649994215 us; error 35 us; skew 500 ppm
I20251028 08:54:09.994598 7037 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:54:09.996500 7037 webserver.cc:492] Webserver started at http://127.4.217.61:43565/ using document root <none> and password file <none>
I20251028 08:54:09.997138 7037 fs_manager.cc:362] Metadata directory not provided
I20251028 08:54:09.997226 7037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:54:10.001955 7037 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20251028 08:54:10.004347 7058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:54:10.005376 7037 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251028 08:54:10.005517 7037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:54:10.006107 7037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:54:10.020329 7037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:54:10.021023 7037 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:54:10.021299 7037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:54:10.039323 7037 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:33257
I20251028 08:54:10.039364 7109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.61:33257 every 8 connection(s)
I20251028 08:54:10.040737 7037 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
I20251028 08:54:10.043012 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 7037
I20251028 08:54:10.048072 7110 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:54:10.051955 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap starting.
I20251028 08:54:10.063310 7110 log.cc:826] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Log is configured to *not* fsync() on all Append() calls
I20251028 08:54:10.096396 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=43 ignored=0} inserts{seen=30 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 08:54:10.096866 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap complete.
I20251028 08:54:10.101737 7110 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:10.102339 7110 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Initialized, Role: FOLLOWER
I20251028 08:54:10.102886 7110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:10.103523 7114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:10.103725 7114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:10.104343 7110 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:54:10.119979 7128 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:10.120133 7128 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: loading cluster ID for follower catalog manager: success
I20251028 08:54:10.124348 7128 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: acquiring CA information for follower catalog manager: success
I20251028 08:54:10.127116 7128 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:54:10.133159 7037 master_runner.cc:186] Error getting master registration for 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111),
I20251028 08:54:10.136583 7037 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:10.189496 5887 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:10.192620 7075 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:42419; Asking this server to re-register.
I20251028 08:54:10.193395 5887 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:10.195122 7075 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:54:10.196249 5753 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:10.198858 7075 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:40773; Asking this server to re-register.
I20251028 08:54:10.199443 5753 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:10.200721 7075 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:54:10.302089 5616 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:10.302238 5616 heartbeater.cc:499] Master 127.4.217.61:33257 was elected leader, sending a full tablet report...
I20251028 08:54:10.303545 7075 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:54303; Asking this server to re-register.
I20251028 08:54:10.304055 5616 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:10.305164 7075 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:54:10.527475 7114 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:10.527710 7114 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:10.529436 7114 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:10.529840 6785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" is_pre_election: true
I20251028 08:54:10.530040 6785 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 7.
W20251028 08:54:10.530047 7059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:10.530551 7060 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters:
I20251028 08:54:10.530921 7114 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251028 08:54:10.531072 7114 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:54:10.531177 7114 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 7 FOLLOWER]: Advancing to term 8
W20251028 08:54:10.531533 7059 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:10.534652 7114 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:10.535207 7114 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 8 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:10.535595 6785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
I20251028 08:54:10.535743 6785 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 7 FOLLOWER]: Advancing to term 8
W20251028 08:54:10.536320 7059 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:10.537936 6785 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 8.
I20251028 08:54:10.538290 7060 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 82b026d06b454d65b27216e3002f29af
I20251028 08:54:10.538554 7114 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Leader election won for term 8
I20251028 08:54:10.538921 7114 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 LEADER]: Becoming Leader. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Running, Role: LEADER
I20251028 08:54:10.539311 7114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:10.541901 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 8 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:10.542135 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:54:10.542728 7140 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:54:10.544957 7140 catalog_manager.cc:679] Loaded metadata for table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:54:10.545332 7140 catalog_manager.cc:679] Loaded metadata for table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]
I20251028 08:54:10.545682 7140 catalog_manager.cc:679] Loaded metadata for table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:54:10.545982 7140 catalog_manager.cc:679] Loaded metadata for table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:54:10.546252 7140 catalog_manager.cc:679] Loaded metadata for table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:54:10.546530 7140 catalog_manager.cc:679] Loaded metadata for table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:54:10.546821 7140 catalog_manager.cc:679] Loaded metadata for table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:54:10.547104 7140 catalog_manager.cc:679] Loaded metadata for table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:54:10.548533 7140 tablet_loader.cc:96] loaded metadata for tablet 2b26ada1ba9546e880d888184f745c40 (table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14])
I20251028 08:54:10.549394 7140 tablet_loader.cc:96] loaded metadata for tablet 6203a2cbeb114d37b8e4d018428c3b0d (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4])
I20251028 08:54:10.549873 7140 tablet_loader.cc:96] loaded metadata for tablet 636688cf451f4e3a97400a1055d1b47b (table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196])
I20251028 08:54:10.550120 7140 tablet_loader.cc:96] loaded metadata for tablet 66fce75a826e48a8b141cf108cb94798 (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508])
I20251028 08:54:10.550388 7140 tablet_loader.cc:96] loaded metadata for tablet 774e1d1fe9334a45a7df75cc5cd098ff (table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2])
I20251028 08:54:10.550693 7140 tablet_loader.cc:96] loaded metadata for tablet 7a5654d71877406f8a5a4e1e9bb08fa4 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad])
I20251028 08:54:10.551702 7140 tablet_loader.cc:96] loaded metadata for tablet e591ce42e8ce4df0952cb231526544ab (table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3])
I20251028 08:54:10.551944 7140 tablet_loader.cc:96] loaded metadata for tablet f0b0ea645082479fa72ede160effe9ea (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e])
I20251028 08:54:10.552134 7140 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:54:10.552896 7140 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:10.552959 7140 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:54:10.554132 7140 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:54:10.554948 7140 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Loaded TSK: 0
I20251028 08:54:10.555704 7140 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:54:10.556481 7140 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:54:10.557287 7140 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:54:10.573807 7075 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b92cfd34a6c4e7fba8f785cc374217f from {username='slave'} at 127.0.0.1:49924
I20251028 08:54:10.574352 7072 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49924:
table { table_name: "default.Table_a90e8e5ec3364d6b80abc305644d234c" } modify_external_catalogs: true
I20251028 08:54:10.574594 7072 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49924:
table { table_name: "default.Table_a90e8e5ec3364d6b80abc305644d234c" } modify_external_catalogs: true
I20251028 08:54:10.574724 7069 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49924:
name: "default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:10.574783 7070 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49924:
name: "default.Table_c496c6ebeb1d44248b341428961d8939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:10.574950 7067 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49924:
name: "default.Table_c896dce32fc04719beb2467b2bf1f855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:10.574200 7074 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49924:
name: "default.Table_6d9d93bde7fa47158f8e7b574783370e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:10.575439 7068 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49924:
table { table_name: "default.Table_976045e19bfe41c3b314c8cc12e020d1" } new_table_name: "default.Table_dfd48b4fd93d407bad513cecd15ecf68" modify_external_catalogs: true
I20251028 08:54:10.576617 7073 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49924:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:54:10.576876 7073 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49924:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:54:10.576591 7071 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49924:
table { table_name: "default.Table_f617dc4ea4f444fa8528dda685e72fa8" } new_table_name: "default.Table_8588dfa71ef54b07996e4bd0a2b29cfc" modify_external_catalogs: true
I20251028 08:54:10.576989 7066 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49924:
table { table_name: "default.Table_64f349518d5d449ea6a147f1795a24fd" } new_table_name: "default.Table_018ba9cb543a4fd0a8e19856261027a1" modify_external_catalogs: true
I20251028 08:54:10.577519 7075 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49924:
name: "default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251028 08:54:10.581215 7127 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f706a3cfdcd google::LogMessage::Fail() at ??:0
@ 0x7f706a3d3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f706a3cf7cc google::LogMessage::Flush() at ??:0
@ 0x7f706a3cfa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f7080d91b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f7080d90411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f7080d9808c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7080d97f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f706d143ee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f707074e6db start_thread at ??:0
@ 0x7f706928071f clone at ??:0
W20251028 08:54:10.754959 5775 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:10.754958 5641 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:10.756170 5895 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.217.61:33257)
W20251028 08:54:10.756641 5895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:10.756805 5895 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:10.757273 5895 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.217.61:33257)
I20251028 08:54:10.888938 7040 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:10.889083 7040 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:54:10.889571 7040 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:54:10.890285 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:54:10.890537 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:54:10.890643 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:10.890884 7040 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:54:11.254015 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 6: this message will repeat every 5th retry.
I20251028 08:54:12.290438 7163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:12.290614 7163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:12.291096 7163 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:12.292897 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:12.293057 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:12.293224 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:12.293444 7163 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251028 08:54:13.014930 7173 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:13.015076 7173 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:13.015666 7173 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:13.017474 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:13.017897 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:13.018033 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:13.018313 7173 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251028 08:54:13.108386 4964 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1hpXWl/build/asan/bin/kudu
/tmp/dist-test-task1hpXWl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--webserver_interface=127.4.217.61
--webserver_port=43565
--builtin_ntp_servers=127.4.217.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251028 08:54:13.419082 7177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 08:54:13.419337 7177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 08:54:13.419413 7177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 08:54:13.426396 7177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 08:54:13.426457 7177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 08:54:13.426510 7177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 08:54:13.426551 7177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 08:54:13.434759 7177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.217.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34547
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.217.61:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.217.61
--webserver_port=43565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 08:43:22 UTC on 24a791456cd2
build id 8756
ASAN enabled
I20251028 08:54:13.436003 7177 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 08:54:13.437559 7177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 08:54:13.444510 7184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:13.444561 7185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 08:54:13.445397 7187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 08:54:13.445479 7177 server_base.cc:1047] running on GCE node
I20251028 08:54:13.446132 7177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 08:54:13.447232 7177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 08:54:13.448419 7177 hybrid_clock.cc:648] HybridClock initialized: now 1761641653448378 us; error 44 us; skew 500 ppm
I20251028 08:54:13.448673 7177 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 08:54:13.450624 7177 webserver.cc:492] Webserver started at http://127.4.217.61:43565/ using document root <none> and password file <none>
I20251028 08:54:13.451259 7177 fs_manager.cc:362] Metadata directory not provided
I20251028 08:54:13.451334 7177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 08:54:13.455693 7177 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 08:54:13.458355 7193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 08:54:13.459481 7177 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 08:54:13.459609 7177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data,/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
uuid: "8b7d4120e0c84998858b0bd4ed9572b7"
format_stamp: "Formatted at 2025-10-28 08:53:41 on dist-test-slave-cmml"
I20251028 08:54:13.460184 7177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 08:54:13.474429 7177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 08:54:13.474999 7177 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 08:54:13.475209 7177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 08:54:13.492515 7177 rpc_server.cc:307] RPC server started. Bound to: 127.4.217.61:33257
I20251028 08:54:13.492563 7244 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.217.61:33257 every 8 connection(s)
I20251028 08:54:13.493932 7177 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/master-1/data/info.pb
I20251028 08:54:13.497272 4964 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1hpXWl/build/asan/bin/kudu as pid 7177
I20251028 08:54:13.500376 7245 sys_catalog.cc:263] Verifying existing consensus state
I20251028 08:54:13.503393 7245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap starting.
I20251028 08:54:13.514420 7245 log.cc:826] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Log is configured to *not* fsync() on all Append() calls
I20251028 08:54:13.535771 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: OK, instance_id { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" instance_seqno: 1761641653488084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251028 08:54:13.536062 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:13.548985 7245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=43 ignored=0} inserts{seen=30 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251028 08:54:13.549446 7245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Bootstrap complete.
I20251028 08:54:13.554076 7245 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:13.554736 7245 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Initialized, Role: FOLLOWER
I20251028 08:54:13.555284 7245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 8.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:13.555858 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:13.556078 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:13.556648 7245 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 08:54:13.580163 7264 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:13.580338 7264 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: loading cluster ID for follower catalog manager: success
I20251028 08:54:13.584553 7264 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: acquiring CA information for follower catalog manager: success
I20251028 08:54:13.587339 7264 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251028 08:54:13.598274 7177 master_runner.cc:186] Error getting master registration for 127.4.217.60:37319: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111),
I20251028 08:54:13.603613 7177 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:13.755874 5753 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:13.758718 7210 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" instance_seqno: 1761641623600675) as {username='slave'} at 127.4.217.2:42851; Asking this server to re-register.
I20251028 08:54:13.759442 5753 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:13.762225 7210 ts_manager.cc:194] Registered new tserver with Master: 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
I20251028 08:54:13.766840 5887 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:13.768666 7210 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" instance_seqno: 1761641624002848) as {username='slave'} at 127.4.217.3:60133; Asking this server to re-register.
I20251028 08:54:13.769508 5887 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:13.770785 7210 ts_manager.cc:194] Registered new tserver with Master: bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455)
I20251028 08:54:13.817056 5616 heartbeater.cc:344] Connected to a master server at 127.4.217.61:33257
I20251028 08:54:13.818321 7210 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "418c8c132d8141ae96893d82e0217aed" instance_seqno: 1761641623178871) as {username='slave'} at 127.4.217.1:53883; Asking this server to re-register.
I20251028 08:54:13.818948 5616 heartbeater.cc:461] Registering TS with master...
I20251028 08:54:13.820287 7210 ts_manager.cc:194] Registered new tserver with Master: 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
I20251028 08:54:14.069379 7272 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:14.069643 7272 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:14.071312 7272 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:14.071666 6785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" is_pre_election: true
I20251028 08:54:14.071872 6785 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 8.
W20251028 08:54:14.071857 7194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:14.072428 7195 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters:
I20251028 08:54:14.072786 7272 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251028 08:54:14.072942 7272 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 08:54:14.073045 7272 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 8 FOLLOWER]: Advancing to term 9
W20251028 08:54:14.073211 7194 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:14.076437 7272 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:14.077004 7272 leader_election.cc:290] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 9 election: Requested vote from peers 0b9421be172b482f9c69d4dc5d2b19a8 (127.4.217.62:41357), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
I20251028 08:54:14.077337 6785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "0b9421be172b482f9c69d4dc5d2b19a8"
I20251028 08:54:14.077525 6785 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 8 FOLLOWER]: Advancing to term 9
W20251028 08:54:14.078269 7194 leader_election.cc:336] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:14.079715 6785 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7d4120e0c84998858b0bd4ed9572b7 in term 9.
I20251028 08:54:14.080070 7195 leader_election.cc:304] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8, 8b7d4120e0c84998858b0bd4ed9572b7; no voters: 82b026d06b454d65b27216e3002f29af
I20251028 08:54:14.080389 7272 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 9 FOLLOWER]: Leader election won for term 9
I20251028 08:54:14.080801 7272 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [term 9 LEADER]: Becoming Leader. State: Replica: 8b7d4120e0c84998858b0bd4ed9572b7, State: Running, Role: LEADER
I20251028 08:54:14.081272 7272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 8.44, Last appended by leader: 44, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:14.083635 7275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 9 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:14.083837 7275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [sys.catalog]: This master's current role is: LEADER
I20251028 08:54:14.084440 7277 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 08:54:14.086578 7277 catalog_manager.cc:679] Loaded metadata for table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508]
I20251028 08:54:14.086917 7277 catalog_manager.cc:679] Loaded metadata for table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2]
I20251028 08:54:14.087182 7277 catalog_manager.cc:679] Loaded metadata for table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad]
I20251028 08:54:14.087450 7277 catalog_manager.cc:679] Loaded metadata for table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e]
I20251028 08:54:14.087674 7277 catalog_manager.cc:679] Loaded metadata for table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196]
I20251028 08:54:14.087913 7277 catalog_manager.cc:679] Loaded metadata for table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3]
I20251028 08:54:14.088142 7277 catalog_manager.cc:679] Loaded metadata for table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4]
I20251028 08:54:14.088382 7277 catalog_manager.cc:679] Loaded metadata for table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14]
I20251028 08:54:14.089793 7277 tablet_loader.cc:96] loaded metadata for tablet 2b26ada1ba9546e880d888184f745c40 (table default.table_f617dc4ea4f444fa8528dda685e72fa8 [id=cc3941670d5b4b32bc09b3ad222c6b14])
I20251028 08:54:14.090687 7277 tablet_loader.cc:96] loaded metadata for tablet 6203a2cbeb114d37b8e4d018428c3b0d (table default.table_d77cbfbaef6940f4bd623eaab2bb30aa [id=c460b691310c4636b32dbfdc5559a5e4])
I20251028 08:54:14.091043 7277 tablet_loader.cc:96] loaded metadata for tablet 636688cf451f4e3a97400a1055d1b47b (table default.table_64f349518d5d449ea6a147f1795a24fd [id=608a411f59d1414e84674bcf38f34196])
I20251028 08:54:14.091224 7277 tablet_loader.cc:96] loaded metadata for tablet 66fce75a826e48a8b141cf108cb94798 (table default.table_976045e19bfe41c3b314c8cc12e020d1 [id=3ada2329e2c7461db5490b9738e68508])
I20251028 08:54:14.091434 7277 tablet_loader.cc:96] loaded metadata for tablet 774e1d1fe9334a45a7df75cc5cd098ff (table default.table_78fd665055cb412390a37c5a185dec85 [id=50d8e788b029409fab90accd8c14c4d2])
I20251028 08:54:14.091650 7277 tablet_loader.cc:96] loaded metadata for tablet 7a5654d71877406f8a5a4e1e9bb08fa4 (table default.table_dd23f4b28a514540926eb05e3f12c357 [id=520f299058714b75b3fe40975a1352ad])
I20251028 08:54:14.092722 7277 tablet_loader.cc:96] loaded metadata for tablet e591ce42e8ce4df0952cb231526544ab (table default.table_7991714d6af648189d6314e21f58740e [id=8a87456e81c34a818f19c585e02dd7e3])
I20251028 08:54:14.093098 7277 tablet_loader.cc:96] loaded metadata for tablet f0b0ea645082479fa72ede160effe9ea (table default.table_a90e8e5ec3364d6b80abc305644d234c [id=56999faebe8a43c48907d91b0444e95e])
I20251028 08:54:14.093369 7277 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 08:54:14.094388 7277 catalog_manager.cc:1269] Loaded cluster ID: 235cc53c0d4a4aeea1549578fb29a1ac
I20251028 08:54:14.094453 7277 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 08:54:14.095520 7277 catalog_manager.cc:1514] Loading token signing keys...
I20251028 08:54:14.096278 7277 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Loaded TSK: 0
I20251028 08:54:14.096936 7277 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 08:54:14.097744 7277 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251028 08:54:14.098569 7277 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251028 08:54:14.177838 7210 master_service.cc:954] ReplaceTablet: received request to replace tablet 2b26ada1ba9546e880d888184f745c40 from {username='slave'} at 127.0.0.1:50678
I20251028 08:54:14.178053 7208 master_service.cc:954] ReplaceTablet: received request to replace tablet 6203a2cbeb114d37b8e4d018428c3b0d from {username='slave'} at 127.0.0.1:50678
I20251028 08:54:14.177861 7209 master_service.cc:954] ReplaceTablet: received request to replace tablet 7ff55e6d288f44428891d6fc7e93e2d2 from {username='slave'} at 127.0.0.1:50678
I20251028 08:54:14.178879 7206 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50678:
name: "default.Table_b01e4320f600411a872c6485f421dd5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:14.179503 7209 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50678:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:54:14.179745 7209 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50678:
table { table_name: "default.Table_7991714d6af648189d6314e21f58740e" } modify_external_catalogs: true
I20251028 08:54:14.179832 7207 master_service.cc:954] ReplaceTablet: received request to replace tablet 896c194b66794478b4fa1d3a08128314 from {username='slave'} at 127.0.0.1:50678
I20251028 08:54:14.180477 7207 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50678:
name: "default.Table_6d9d93bde7fa47158f8e7b574783370e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:14.180949 7205 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50678:
table { table_name: "default.Table_dd23f4b28a514540926eb05e3f12c357" } new_table_name: "default.Table_72938f3a321a4a74ba29b7aace4a1998" modify_external_catalogs: true
I20251028 08:54:14.182387 7201 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50678:
table { table_name: "default.Table_a90e8e5ec3364d6b80abc305644d234c" } modify_external_catalogs: true
I20251028 08:54:14.182476 7203 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50678:
name: "default.Table_c496c6ebeb1d44248b341428961d8939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 08:54:14.182713 7201 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50678:
table { table_name: "default.Table_a90e8e5ec3364d6b80abc305644d234c" } modify_external_catalogs: true
I20251028 08:54:14.182981 7202 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50678:
table { table_name: "default.Table_f617dc4ea4f444fa8528dda685e72fa8" } new_table_name: "default.Table_8588dfa71ef54b07996e4bd0a2b29cfc" modify_external_catalogs: true
I20251028 08:54:14.183024 7204 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50678:
table { table_name: "default.Table_d77cbfbaef6940f4bd623eaab2bb30aa" } new_table_name: "default.Table_a32163429a914f28ad8f6bac7f7c11c0" modify_external_catalogs: true
I20251028 08:54:14.186446 6785 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Refusing update from remote peer 8b7d4120e0c84998858b0bd4ed9572b7: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 9 index: 46. (index mismatch)
W20251028 08:54:14.186569 7194 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 -> Peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Couldn't send request to peer 82b026d06b454d65b27216e3002f29af. Status: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251028 08:54:14.187117 7275 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 43, Time since last communication: 0.000s
F20251028 08:54:14.187638 7263 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 9 received events: NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_976045e19bfe41c3b314c8cc12e020d1, message={"server":"","servicePrincipal":"","db":"default","table":"table_976045e19bfe41c3b314c8cc12e020d1","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_976045e19bfe41c3b314c8cc12e020d1\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_976045e19bfe41c3b314c8cc12e020d1\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_976045e19bfe41c3b314c8cc12e020d1\",\"kudu.table_id\":\"3ada2329e2c7461db5490b9738e68508\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761641624, eventType=CREATE_TABLE, dbName=default, tableName=table_a5ab4d2da1704bf4a0c944f5c59dc3f7, message={"server":"","servicePrincipal":"","db":"default","table":"table_a5ab4d2da1704bf4a0c944f5c59dc3f7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761641624},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/warehouse/table_a5ab4d2da1704bf4a0c944f5c59dc3f7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a5ab4d2da1704bf4a0c944f5c59dc3f7\",\"kudu.table_id\":\"419efc64c9894b7dbac23a5c4d09e3b7\",\"transient_lastDdlTime\":\"1761641624\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"235cc53c0d4a4aeea1549578fb29a1ac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761641624,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f4f54cd7dcd google::LogMessage::Fail() at ??:0
@ 0x7f4f54cdbb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4f54cd77cc google::LogMessage::Flush() at ??:0
@ 0x7f4f54cd7a11 google::LogMessage::~LogMessage() at ??:0
I20251028 08:54:14.200439 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7d4120e0c84998858b0bd4ed9572b7. Latest consensus state: current_term: 9 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:14.200731 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:14.204684 7210 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Sending DeleteTablet for 3 replicas of tablet 2b26ada1ba9546e880d888184f745c40
I20251028 08:54:14.205130 7208 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8b7d4120e0c84998858b0bd4ed9572b7: Sending DeleteTablet for 3 replicas of tablet 6203a2cbeb114d37b8e4d018428c3b0d
@ 0x7f4f6b699b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251028 08:54:14.206545 7208 catalog_manager.cc:6719] ReplaceTablet: tablet 6203a2cbeb114d37b8e4d018428c3b0d deleted and replaced by tablet d8ee467598b04b2ba26390a30a879b7e
I20251028 08:54:14.216320 7210 catalog_manager.cc:6719] ReplaceTablet: tablet 2b26ada1ba9546e880d888184f745c40 deleted and replaced by tablet 6c6274bbe61b4ac296c35fdeeb4568c0
@ 0x7f4f6b698411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f4f6b6a008c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f4f6b69ff09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251028 08:54:14.237248 5686 tablet_service.cc:1552] Processing DeleteTablet for tablet 6203a2cbeb114d37b8e4d018428c3b0d with delete_type TABLET_DATA_DELETED (replaced by tablet d8ee467598b04b2ba26390a30a879b7e) from {username='slave'} at 127.0.0.1:47112
I20251028 08:54:14.237253 5684 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b26ada1ba9546e880d888184f745c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c6274bbe61b4ac296c35fdeeb4568c0) from {username='slave'} at 127.0.0.1:47112
@ 0x7f4f57a4bee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4f5b0566db start_thread at ??:0
I20251028 08:54:14.245198 5818 tablet_service.cc:1552] Processing DeleteTablet for tablet 6203a2cbeb114d37b8e4d018428c3b0d with delete_type TABLET_DATA_DELETED (replaced by tablet d8ee467598b04b2ba26390a30a879b7e) from {username='slave'} at 127.0.0.1:54318
I20251028 08:54:14.245520 5820 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b26ada1ba9546e880d888184f745c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c6274bbe61b4ac296c35fdeeb4568c0) from {username='slave'} at 127.0.0.1:54318
I20251028 08:54:14.248160 5544 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b26ada1ba9546e880d888184f745c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c6274bbe61b4ac296c35fdeeb4568c0) from {username='slave'} at 127.0.0.1:52140
I20251028 08:54:14.248510 5546 tablet_service.cc:1552] Processing DeleteTablet for tablet 6203a2cbeb114d37b8e4d018428c3b0d with delete_type TABLET_DATA_DELETED (replaced by tablet d8ee467598b04b2ba26390a30a879b7e) from {username='slave'} at 127.0.0.1:52140
@ 0x7f4f53b8871f clone at ??:0
I20251028 08:54:14.258126 7295 tablet_replica.cc:333] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:54:14.258388 7295 raft_consensus.cc:2243] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 08:54:14.258656 7295 raft_consensus.cc:2272] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:54:14.259373 7295 ts_tablet_manager.cc:1916] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:54:14.260332 7294 tablet_replica.cc:333] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:54:14.260532 7294 raft_consensus.cc:2243] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 08:54:14.260766 7294 raft_consensus.cc:2272] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:54:14.261338 7294 ts_tablet_manager.cc:1916] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:54:14.262048 7296 tablet_replica.cc:333] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
I20251028 08:54:14.262218 7296 raft_consensus.cc:2243] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 08:54:14.262427 7296 raft_consensus.cc:2272] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:54:14.263072 7296 ts_tablet_manager.cc:1916] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:54:14.268028 7295 ts_tablet_manager.cc:1929] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:54:14.268149 7295 log.cc:1199] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/6203a2cbeb114d37b8e4d018428c3b0d
I20251028 08:54:14.268637 7295 ts_tablet_manager.cc:1950] T 6203a2cbeb114d37b8e4d018428c3b0d P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
I20251028 08:54:14.269878 7295 tablet_replica.cc:333] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: stopping tablet replica
I20251028 08:54:14.270144 7295 raft_consensus.cc:2243] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 08:54:14.270332 7295 raft_consensus.cc:2272] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:54:14.270947 7295 ts_tablet_manager.cc:1916] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:54:14.271499 7294 ts_tablet_manager.cc:1929] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:54:14.271572 7294 log.cc:1199] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/6203a2cbeb114d37b8e4d018428c3b0d
I20251028 08:54:14.271875 7294 ts_tablet_manager.cc:1950] T 6203a2cbeb114d37b8e4d018428c3b0d P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
I20251028 08:54:14.279176 7295 ts_tablet_manager.cc:1929] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:54:14.279326 7295 log.cc:1199] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-2/wal/wals/2b26ada1ba9546e880d888184f745c40
I20251028 08:54:14.279822 7295 ts_tablet_manager.cc:1950] T 2b26ada1ba9546e880d888184f745c40 P bbab3d08643e49e786c2174faaa741d9: Deleting consensus metadata
I20251028 08:54:14.282044 7296 ts_tablet_manager.cc:1929] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:54:14.282146 7296 log.cc:1199] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/2b26ada1ba9546e880d888184f745c40
I20251028 08:54:14.282595 7296 ts_tablet_manager.cc:1950] T 2b26ada1ba9546e880d888184f745c40 P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:54:14.272615 7294 tablet_replica.cc:333] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: stopping tablet replica
I20251028 08:54:14.284062 7294 raft_consensus.cc:2243] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 LEADER]: Raft consensus shutting down.
I20251028 08:54:14.284081 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:14.284363 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:14.284516 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } } }
I20251028 08:54:14.284807 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251028 08:54:14.285303 7294 raft_consensus.cc:2272] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:54:14.285869 7294 ts_tablet_manager.cc:1916] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:54:14.286288 7296 tablet_replica.cc:333] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: stopping tablet replica
I20251028 08:54:14.286487 7296 raft_consensus.cc:2243] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 1 LEADER]: Raft consensus shutting down.
I20251028 08:54:14.286809 7296 raft_consensus.cc:2272] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 08:54:14.287196 7296 ts_tablet_manager.cc:1916] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251028 08:54:14.292096 7296 ts_tablet_manager.cc:1929] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:54:14.292208 7296 log.cc:1199] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-0/wal/wals/6203a2cbeb114d37b8e4d018428c3b0d
I20251028 08:54:14.292583 7296 ts_tablet_manager.cc:1950] T 6203a2cbeb114d37b8e4d018428c3b0d P 418c8c132d8141ae96893d82e0217aed: Deleting consensus metadata
I20251028 08:54:14.293457 7294 ts_tablet_manager.cc:1929] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251028 08:54:14.293570 7294 log.cc:1199] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting WAL directory at /tmp/dist-test-task1hpXWl/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761641606079212-4964-0/minicluster-data/ts-1/wal/wals/2b26ada1ba9546e880d888184f745c40
I20251028 08:54:14.294000 7294 ts_tablet_manager.cc:1950] T 2b26ada1ba9546e880d888184f745c40 P 52a17de1c37d4a339d1fcb35b550aad7: Deleting consensus metadata
W20251028 08:54:14.438652 5501 connection.cc:537] server connection from 127.0.0.1:52140 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.438625 5774 connection.cc:537] server connection from 127.0.0.1:54318 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.438623 5640 connection.cc:537] server connection from 127.0.0.1:47112 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.438691 5775 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.438897 5503 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.439076 6762 connection.cc:537] server connection from 127.0.0.1:35266 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.439154 5641 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.439504 5895 connection.cc:537] client connection to 127.4.217.61:33257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 08:54:14.440141 5895 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.217.61:33257)
W20251028 08:54:14.440723 5895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:14.440903 5895 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.217.61:33257)
W20251028 08:54:14.441419 5895 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.217.61:33257)
I20251028 08:54:14.703411 7284 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:14.703550 7284 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:14.704154 7284 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:14.706092 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:14.706220 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:14.706353 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:14.706553 7284 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:14.767483 7274 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:14.767632 7274 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:14.768182 7274 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:14.768991 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:54:14.769196 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:54:14.769309 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:14.769574 7274 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:15.260198 7316 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:15.260371 7316 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:15.261037 7316 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:15.261631 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20251028 08:54:15.262684 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:15.262801 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:15.262962 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:15.263163 7316 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:15.828570 4964 master-stress-test.cc:577] Killing test thread 1/25
I20251028 08:54:16.040696 7325 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:16.040855 7325 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:16.041365 7325 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:16.043447 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:16.043753 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:16.043876 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:16.044098 7325 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:16.723960 7337 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:16.724139 7337 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:16.724649 7337 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:16.726349 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:16.726519 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:16.726639 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:16.726874 7337 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:17.229770 7344 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:17.229924 7344 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:17.230521 7344 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:17.232177 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:17.232339 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:17.232450 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:17.232671 7344 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:18.538305 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:18.538460 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:19.607848 7366 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:19.608039 7366 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:19.608628 7366 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:19.611281 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:19.613783 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:19.613914 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:19.614153 7366 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:20.860386 7376 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:20.860584 7376 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:20.861407 7376 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:20.862174 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:54:20.862342 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:54:20.862416 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:20.862623 7376 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:21.536495 7382 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:21.536733 7382 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
W20251028 08:54:21.538864 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251028 08:54:21.538889 7382 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:21.539973 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:21.540206 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:21.540331 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:21.540549 7382 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251028 08:54:21.771030 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 11: this message will repeat every 5th retry.
I20251028 08:54:23.541388 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:23.541548 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:23.980820 7401 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:23.980978 7401 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:23.981570 7401 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:23.982394 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:23.982582 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:23.982695 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:23.982934 7401 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:24.574792 7408 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:24.574972 7408 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:24.575539 7408 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:24.577472 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:24.577652 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:24.577783 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:24.578011 7408 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:26.908428 7431 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:26.908569 7431 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:54:26.909053 7431 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:54:26.909698 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:54:26.909871 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:54:26.909941 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:26.910143 7431 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:28.543689 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:28.543833 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:28.880537 7445 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:28.880731 7445 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:28.881223 7445 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:28.881948 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:54:28.882171 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:54:28.882282 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:28.882504 7445 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:29.579924 7452 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:29.580089 7452 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:29.580578 7452 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:29.581262 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:29.581439 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:29.581530 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:29.581776 7452 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:54:32.226634 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 16: this message will repeat every 5th retry.
I20251028 08:54:33.546108 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:33.546267 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:36.136240 7497 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:36.136430 7497 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:36.137154 7497 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:36.138016 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:54:36.138182 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:54:36.138288 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:36.138512 7497 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:37.215346 7508 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:37.215510 7508 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:37.215981 7508 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:37.216720 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:54:37.216951 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:54:37.217038 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:37.217295 7508 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:37.279798 7512 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:37.279958 7512 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:37.280541 7512 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:37.281282 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251028 08:54:37.282528 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:37.282646 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:37.282770 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:37.282994 7512 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:38.548938 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:38.549119 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:54:42.694850 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 21: this message will repeat every 5th retry.
I20251028 08:54:42.949199 7552 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:42.949365 7552 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:42.949962 7552 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:42.950803 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:42.950987 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:42.951090 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:42.951292 7552 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:43.551329 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:43.551484 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:54:44.311913 5753 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (77 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:54:44.358902 7568 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:44.359058 7568 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:44.359622 7568 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:44.360352 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:54:44.360574 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:54:44.360664 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:44.360869 7568 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:54:44.493364 5616 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (60 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:44.583690 5885 heartbeater.cc:646] Failed to heartbeat to 127.4.217.60:37319 (81 consecutive failures): Network error: Failed to ping master at 127.4.217.60:37319: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:45.508086 7572 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:45.508212 7572 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:54:45.508680 7572 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:54:45.509306 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:54:45.509472 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:54:45.509573 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:45.509824 7572 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:46.465544 7579 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:46.465699 7579 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:46.466209 7579 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:46.466912 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:54:46.467106 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:54:46.467180 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:46.467372 7579 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:46.651685 7582 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:46.651870 7582 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:46.652477 7582 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:46.653350 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:54:46.654397 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:54:46.654543 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:54:46.654655 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:46.654892 7582 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:47.223344 7579 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:47.223526 7579 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:47.224079 7579 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:47.224754 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:47.224928 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:47.225008 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:47.225199 7579 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:48.553503 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:48.553687 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:54:52.707887 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 26: this message will repeat every 5th retry.
I20251028 08:54:53.555866 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:53.556012 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:54.419209 7650 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:54.419373 7650 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:54.420071 7650 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:54.420893 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:54:54.422099 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:54.422245 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:54.422394 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:54.422622 7650 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:56.897908 7669 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:56.898100 7669 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:56.898947 7669 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:56.899919 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:54:56.900134 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:54:56.900206 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:56.900405 7669 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:57.688584 7679 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:54:57.688784 7679 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:54:57.689371 7679 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:54:57.691411 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:54:57.691576 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:54:57.691712 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:54:57.691951 7679 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:54:58.110486 7669 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:58.110697 7669 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:54:58.111153 7669 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:54:58.111886 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:54:58.112066 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:54:58.112169 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:58.112437 7669 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:54:58.558100 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:54:58.558269 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:54:58.953723 7669 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:54:58.953876 7669 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:54:58.954362 7669 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:54:58.955137 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:54:58.955319 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:54:58.955389 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:54:58.955602 7669 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:00.514668 7700 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:00.514844 7700 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:00.515355 7700 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:00.516119 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:55:00.516268 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:55:00.516340 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:00.516543 7700 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:55:01.426182 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 31: this message will repeat every 5th retry.
I20251028 08:55:03.560341 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:03.560534 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:05.585707 7742 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:05.585866 7742 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:05.586467 7742 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:05.587119 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:55:05.587316 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:55:05.587391 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:05.587610 7742 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:05.752090 7742 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:05.752238 7742 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:05.752871 7742 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:05.753561 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:55:05.753787 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:55:05.753893 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:05.754161 7742 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:08.562518 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:08.562654 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:55:11.436194 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 36: this message will repeat every 5th retry.
I20251028 08:55:13.564934 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:13.565122 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:15.861827 7816 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:55:15.861990 7816 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:55:15.862918 7816 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:55:15.863785 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251028 08:55:15.864948 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:55:15.865092 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:55:15.865195 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:55:15.865404 7816 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:55:17.227984 7829 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:17.228143 7829 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:55:17.228753 7829 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:55:17.229363 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:55:17.229640 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:55:17.229769 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:17.230015 7829 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:18.159097 7836 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:18.159250 7836 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:18.160000 7836 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:18.160746 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:55:18.160913 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:55:18.161017 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:18.161345 7836 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:18.567251 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:18.567425 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:20.897048 7863 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:20.897217 7863 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:20.897831 7863 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:20.898708 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:55:20.898885 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:55:20.898955 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:20.899238 7863 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:55:21.495050 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 41: this message will repeat every 5th retry.
I20251028 08:55:23.569815 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:23.569989 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:23.890900 7885 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:23.891054 7885 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:23.891585 7885 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:23.892436 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:55:23.892674 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:55:23.892791 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:23.893045 7885 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:25.480232 7894 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:25.480389 7894 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:25.480983 7894 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:25.481709 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:55:25.481904 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:55:25.481972 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:25.482160 7894 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:26.990103 7907 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:55:26.990288 7907 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:55:26.991041 7907 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:55:26.991865 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:55:26.992967 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:55:26.993181 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:55:26.993306 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:55:26.993881 7907 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:55:28.572214 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:28.572435 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:29.319869 7931 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:29.320020 7931 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:29.320601 7931 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:29.321384 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:55:29.321540 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:55:29.321684 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:29.321884 7931 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:31.193996 7940 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:31.194159 7940 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:55:31.194679 7940 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:55:31.195359 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:55:31.195569 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:55:31.195657 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:31.195886 7940 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:55:31.916612 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 46: this message will repeat every 5th retry.
I20251028 08:55:33.574717 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:33.574869 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:34.905109 7971 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:34.905279 7971 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:34.906006 7971 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:34.906711 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:55:34.906915 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:55:34.907006 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:34.907192 7971 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:35.441098 7978 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:55:35.441244 7978 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:55:35.441808 7978 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:55:35.442523 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:55:35.443873 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:55:35.444028 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:55:35.444157 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:55:35.444352 7978 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:55:38.577039 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:38.577188 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:39.200726 8009 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:55:39.200886 8009 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:55:39.201359 8009 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:55:39.203979 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:55:39.204109 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:55:39.204253 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:55:39.204537 8009 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:55:41.254105 8028 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:41.254264 8028 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:41.254806 8028 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:41.255461 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:55:41.255630 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:55:41.255707 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:41.255954 8028 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:41.272744 8028 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:41.272851 8028 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:41.273370 8028 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:41.273962 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:55:41.274111 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:55:41.274184 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:41.274374 8028 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:55:41.531905 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 51: this message will repeat every 5th retry.
I20251028 08:55:43.579538 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:43.579715 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:55:44.485095 5753 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (225 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:55:44.607450 5887 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (180 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:55:44.607462 5616 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (180 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:55:44.762267 8062 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:44.762506 8062 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:55:44.763295 8062 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:55:44.764113 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:55:44.764358 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:55:44.764464 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:44.764703 8062 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:45.356352 8062 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:45.356499 8062 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:45.357036 8062 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:45.357746 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:55:45.357914 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:55:45.358026 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:45.358227 8062 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:46.532966 8070 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:46.533109 8070 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:46.533674 8070 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:46.534437 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:55:46.534636 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:55:46.534715 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:46.534942 8070 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:46.753996 8073 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:55:46.754185 8073 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:55:46.754805 8073 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:55:46.755540 6759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251028 08:55:46.756609 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:55:46.756801 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:55:46.756927 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:55:46.757148 8073 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:55:48.582031 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:48.582203 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:50.238714 8106 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:50.238847 8106 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:50.239538 8106 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:50.240265 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:55:50.240437 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:55:50.240520 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:50.240751 8106 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:55:51.989907 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 56: this message will repeat every 5th retry.
I20251028 08:55:53.584640 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:53.584836 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:55:55.957003 8149 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:55.957149 8149 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:55.957727 8149 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:55.958384 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:55:55.958570 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:55:55.958638 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:55.958817 8149 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:57.220640 8159 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:55:57.220793 8159 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:55:57.221329 8159 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:55:57.222074 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:55:57.222277 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:55:57.222401 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:55:57.222618 8159 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:55:57.464227 8165 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:55:57.464375 8165 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:55:57.464986 8165 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:55:57.465813 6759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:55:57.466935 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:55:57.467113 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:55:57.467232 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:55:57.467418 8165 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:55:58.587049 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:55:58.587185 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:56:01.348758 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 61: this message will repeat every 5th retry.
I20251028 08:56:02.626817 8213 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:02.626986 8213 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:02.627614 8213 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:02.628459 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:56:02.628685 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:56:02.628783 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:02.629036 8213 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:03.177368 8213 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:03.177536 8213 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:03.178066 8213 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:03.178745 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:56:03.178893 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:56:03.178978 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:03.179162 8213 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:03.353847 8213 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:03.354014 8213 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:56:03.354619 8213 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:56:03.355337 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:56:03.355531 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:56:03.355605 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:03.355823 8213 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:03.480222 6764 maintenance_manager.cc:419] P 0b9421be172b482f9c69d4dc5d2b19a8: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033349
I20251028 08:56:03.517483 6763 maintenance_manager.cc:643] P 0b9421be172b482f9c69d4dc5d2b19a8: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.037s user 0.020s sys 0.000s Metrics: {"bytes_written":14067,"cfile_init":1,"dirs.queue_time_us":361,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":980,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":534,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":32,"thread_start_us":490,"threads_started":2,"wal-append.queue_time_us":367}
I20251028 08:56:03.518523 6764 maintenance_manager.cc:419] P 0b9421be172b482f9c69d4dc5d2b19a8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1543 bytes on disk
I20251028 08:56:03.519085 6763 maintenance_manager.cc:643] P 0b9421be172b482f9c69d4dc5d2b19a8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20251028 08:56:03.590233 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:03.590437 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:05.123939 8236 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:56:05.124135 8236 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:56:05.124830 8236 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:56:05.125694 6759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:56:05.126837 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:56:05.126976 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:56:05.127103 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:56:05.127319 8236 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:56:08.592536 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:08.592687 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:56:11.212993 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 66: this message will repeat every 5th retry.
I20251028 08:56:12.341003 8295 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:12.341171 8295 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:12.341918 8295 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:12.342599 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:56:12.342873 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:56:12.343001 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:12.343225 8295 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:12.700953 8295 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:12.701120 8295 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:56:12.701639 8295 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:56:12.702185 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:56:12.702369 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:56:12.702443 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:12.702664 8295 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:13.595129 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:13.595283 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:15.048808 8309 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:15.048950 8309 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:15.049500 8309 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:15.050242 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:56:15.050403 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:56:15.050484 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:15.050721 8309 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:18.481861 8335 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:56:18.482040 8335 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:56:18.482609 8335 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:56:18.483361 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:56:18.484377 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:56:18.484520 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:56:18.484611 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:56:18.484803 8335 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:56:18.597158 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:18.597329 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:19.098496 8339 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:19.098647 8339 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:19.099196 8339 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:19.100023 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:56:19.100265 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:56:19.100356 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:19.100601 8339 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:56:21.365237 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 71: this message will repeat every 5th retry.
I20251028 08:56:22.199093 8372 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:22.199259 8372 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:56:22.199764 8372 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:56:22.200471 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:56:22.200646 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:56:22.200803 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:22.201064 8372 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:23.599500 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:23.599649 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:28.601976 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:28.602136 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:28.733134 8449 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:28.733316 8449 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:28.733999 8449 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:28.734797 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:56:28.735000 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:56:28.735081 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:28.735347 8449 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:31.114495 8468 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:31.114634 8468 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:31.115213 8468 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:31.116026 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:56:31.116227 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:56:31.116353 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:31.116576 8468 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:31.249586 8477 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:56:31.249827 8477 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:56:31.250478 8477 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:56:31.251340 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:56:31.252629 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:56:31.252787 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:56:31.252884 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:56:31.253137 8477 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251028 08:56:31.597287 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 76: this message will repeat every 5th retry.
I20251028 08:56:33.603885 8495 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:33.604061 8495 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:33.604265 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:33.604455 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:33.604687 8495 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:33.605434 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:56:33.605684 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:56:33.605769 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:33.605963 8495 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:34.075723 8495 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:34.075876 8495 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:56:34.076455 8495 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:56:34.077184 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:56:34.077373 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:56:34.077476 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:34.077729 8495 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:36.895007 8530 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:36.895171 8530 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:36.895733 8530 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:36.896468 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:56:36.896649 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:56:36.896721 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:36.896924 8530 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:38.606876 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:38.607048 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:56:41.116304 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 81: this message will repeat every 5th retry.
I20251028 08:56:42.521381 8605 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:42.521544 8605 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:56:42.522132 8605 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:56:42.522918 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:56:42.523082 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:56:42.523164 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:42.523342 8605 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:42.631824 8605 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:42.631990 8605 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:42.632500 8605 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:42.633332 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:56:42.633524 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:56:42.633641 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:42.633849 8605 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:43.609381 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:43.609575 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:56:44.611811 5753 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (381 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:56:44.723196 5616 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (300 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:56:44.723194 5887 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (300 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:56:48.492059 8666 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:48.492190 8666 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:56:48.492790 8666 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:56:48.493428 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:56:48.493618 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:56:48.493737 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:48.493943 8666 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:48.612051 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:48.612187 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:50.568722 8691 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:56:50.568885 8691 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:56:50.569766 8691 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:56:50.570573 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:56:50.572088 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:56:50.572265 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:56:50.572362 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:56:50.572582 8691 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251028 08:56:50.756371 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 86: this message will repeat every 5th retry.
I20251028 08:56:52.082494 8702 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:52.082638 8702 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:52.083099 8702 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:52.083815 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:56:52.084035 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:56:52.084122 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:52.084339 8702 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:52.266052 8702 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:52.266214 8702 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:56:52.266777 8702 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:56:52.267604 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:56:52.267796 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:56:52.267903 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:52.268121 8702 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:52.544503 8702 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:52.544692 8702 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:52.545328 8702 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:52.546216 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:56:52.546417 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:56:52.546502 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:52.546732 8702 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:52.824682 8702 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:52.824837 8702 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:52.825345 8702 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:52.826143 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:56:52.826357 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:56:52.826462 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:52.826681 8702 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:53.614399 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:53.614533 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:56:56.089596 8750 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:56:56.089748 8750 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:56:56.090265 8750 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:56:56.091406 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:56:56.091612 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:56:56.091706 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:56:56.091915 8750 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:56:58.616585 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:56:58.616739 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:01.812693 8817 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:01.812850 8817 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:01.813409 8817 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:01.814227 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:01.814424 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:01.814523 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:01.814746 8817 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:57:01.828568 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 91: this message will repeat every 5th retry.
I20251028 08:57:02.721781 8828 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:02.721951 8828 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:02.722527 8828 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:02.723249 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:57:02.723481 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:57:02.723560 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:02.723740 8828 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:02.987305 8837 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:02.987479 8837 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:02.988148 8837 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:02.988998 6759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:57:02.989992 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:02.990134 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:57:02.990285 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:02.990504 8837 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:03.618954 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:03.619144 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:06.794410 8880 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:06.794557 8880 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:06.795152 8880 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:06.795948 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:57:06.796119 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:57:06.796192 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:06.796414 8880 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:08.621311 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:08.621503 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:08.876286 8907 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:08.876428 8907 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:57:08.876868 8907 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:57:08.877676 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:57:08.877867 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:57:08.877952 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:08.878186 8907 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:57:11.614554 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 96: this message will repeat every 5th retry.
I20251028 08:57:12.143193 8942 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:12.143362 8942 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:12.143999 8942 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:12.144747 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:57:12.145972 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:12.146215 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:57:12.146343 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:12.146574 8942 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:12.201661 8946 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:12.201812 8946 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:57:12.202419 8946 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:57:12.203177 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:57:12.203383 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:57:12.203486 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:12.203732 8946 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:13.225792 8960 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:13.225975 8960 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:13.226491 8960 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:13.228365 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:13.228509 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:57:13.228645 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:13.228880 8960 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:13.623620 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:13.623822 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:15.618026 8988 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:15.618197 8988 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:15.618808 8988 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:15.623188 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:57:15.623371 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:57:15.623497 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:15.623721 8988 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:16.759277 9000 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:16.759500 9000 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:16.760042 9000 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:16.760780 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:57:16.760960 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:57:16.761022 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:16.761250 9000 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:17.655422 9014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:17.655618 9014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:17.656224 9014 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:17.656961 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251028 08:57:17.658298 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:17.658439 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:57:17.658546 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:17.658743 9014 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:18.626220 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:18.626394 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:19.363759 9039 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:19.363919 9039 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:19.364475 9039 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:19.365152 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:19.365335 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:19.365419 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:19.365634 9039 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:19.948652 9039 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:19.948801 9039 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:19.949280 9039 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:19.950040 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:57:19.950273 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:57:19.950353 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:19.950589 9039 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:57:21.299360 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 101: this message will repeat every 5th retry.
I20251028 08:57:23.628811 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:23.628974 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:25.661852 9090 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:25.661999 9090 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:25.662727 9090 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:25.663511 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:25.663693 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:25.663787 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:25.663995 9090 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:28.631398 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:28.631564 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:28.886853 9138 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:28.887059 9138 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:57:28.887960 9138 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:57:28.888442 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:57:28.888625 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:57:28.888700 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:28.888937 9138 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:29.966682 9147 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:29.966842 9147 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:29.967425 9147 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:29.968264 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:29.968473 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:29.968573 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:29.968793 9147 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:30.936682 9155 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:30.936839 9155 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:30.937459 9155 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:30.938131 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:57:30.938364 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:57:30.938441 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:30.938673 9155 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:31.409673 9155 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:31.409821 9155 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:31.410317 9155 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:31.411268 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:57:31.411450 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:57:31.411556 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:31.411783 9155 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:57:32.219866 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 106: this message will repeat every 5th retry.
I20251028 08:57:33.633936 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:33.634193 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:35.348348 9201 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:35.348553 9201 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:35.349351 9201 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:35.350304 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:57:35.351516 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:35.351672 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:57:35.351753 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:35.351965 9201 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:38.309123 9236 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:38.309259 9236 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:38.309774 9236 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:38.310406 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:57:38.310631 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:57:38.310707 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:38.310884 9236 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:38.636430 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:38.636618 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:39.087037 9246 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:39.087177 9246 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:39.087656 9246 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:39.088361 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:39.088575 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:39.088696 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:39.088972 9246 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:39.285444 9246 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:39.285609 9246 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:57:39.286098 9246 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:57:39.286809 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:57:39.286970 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:57:39.287065 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:39.287266 9246 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:57:41.723738 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 111: this message will repeat every 5th retry.
I20251028 08:57:41.726625 9274 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:41.726769 9274 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:41.727448 9274 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:41.728086 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:57:41.728260 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:57:41.728343 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:41.728552 9274 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:41.999910 9276 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:42.000082 9276 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:42.001137 9276 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:42.002305 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:57:42.006560 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:42.006739 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:57:42.006825 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:42.007016 9276 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:42.231491 9274 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:42.231643 9274 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:42.232156 9274 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:42.232785 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:42.232986 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:42.233067 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:42.233268 9274 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:43.639276 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:43.639479 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:57:44.848021 5616 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (420 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:44.848059 5887 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (420 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:57:45.074452 5751 heartbeater.cc:646] Failed to heartbeat to 127.4.217.60:37319 (557 consecutive failures): Network error: Failed to ping master at 127.4.217.60:37319: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:57:46.811465 9334 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:57:46.811682 9334 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:57:46.812260 9334 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:57:46.814131 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:57:46.814280 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:57:46.814427 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:57:46.814656 9334 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:57:46.935672 9338 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:46.935868 9338 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:46.936507 9338 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:46.937254 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:57:46.937445 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:57:46.937529 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:46.937768 9338 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:47.688472 9338 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:47.688650 9338 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:57:47.689311 9338 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:57:47.690132 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:57:47.690320 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:57:47.690399 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:47.690623 9338 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:48.641861 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:48.642011 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:50.623916 9372 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:50.624050 9372 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:50.624538 9372 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:50.625201 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:50.625419 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:50.625520 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:50.625787 9372 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:51.161182 9372 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:51.161321 9372 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:51.161829 9372 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:51.162465 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:57:51.162683 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:57:51.162755 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:51.162994 9372 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:57:52.146878 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 116: this message will repeat every 5th retry.
I20251028 08:57:53.644534 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:53.644708 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:57:56.008185 9421 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:56.008359 9421 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:57:56.009045 9421 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:57:56.009864 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:57:56.010077 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:57:56.010180 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:56.010430 9421 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:57.717996 9449 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:57:57.718169 9449 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:57:57.718803 9449 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:57:57.719547 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:57:57.719712 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:57:57.719779 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:57:57.719996 9449 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:57:58.647719 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:57:58.647933 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:00.629365 9480 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:00.629495 9480 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:00.630009 9480 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:00.630790 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:58:00.630975 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:58:00.631098 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:00.631309 9480 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:58:01.882246 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 121: this message will repeat every 5th retry.
I20251028 08:58:02.296712 9500 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:02.296875 9500 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:02.297448 9500 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:02.298326 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:58:02.298506 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:58:02.298573 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:02.298774 9500 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:03.521194 9514 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:58:03.521384 9514 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:58:03.522172 9514 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:58:03.522974 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251028 08:58:03.524322 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:58:03.524499 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:58:03.524619 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:58:03.524878 9514 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:58:03.649834 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:03.649998 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:05.788329 9538 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:05.788524 9538 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:58:05.789116 9538 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:58:05.790055 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:58:05.790251 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:58:05.790360 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:05.790584 9538 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:08.652320 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:08.652495 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:10.451424 9587 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:10.451586 9587 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:10.452152 9587 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:10.452872 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:58:10.453076 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:58:10.453152 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:10.453397 9587 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:11.385387 9595 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:11.385546 9595 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:11.386133 9595 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:11.386894 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:58:11.387159 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:58:11.387257 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:11.387498 9595 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:11.961776 9595 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:11.961932 9595 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:11.962527 9595 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:11.963197 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:58:11.963407 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:58:11.963514 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:11.963768 9595 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:58:12.185882 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 126: this message will repeat every 5th retry.
I20251028 08:58:13.654593 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:13.654776 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:15.853334 9642 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:15.853493 9642 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:15.854351 9642 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:15.855082 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:58:15.855278 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:58:15.855391 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:15.855634 9642 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:18.656965 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:18.657099 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:58:22.073714 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 131: this message will repeat every 5th retry.
I20251028 08:58:22.976390 9717 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:58:22.976570 9717 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:58:22.977329 9717 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:58:22.978117 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:58:22.982473 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:58:22.982637 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:58:22.982815 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:58:22.983058 9717 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:58:23.659775 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:23.659924 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:23.673950 9729 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:23.674094 9729 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:23.674587 9729 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:23.675271 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:58:23.675457 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:58:23.675527 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:23.675742 9729 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:24.121867 9729 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:24.122032 9729 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:58:24.122571 9729 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:58:24.123340 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:58:24.123515 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:58:24.123601 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:24.123788 9729 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:27.550706 9761 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:27.550849 9761 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:27.551437 9761 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:27.552050 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:58:27.552260 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:58:27.552371 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:27.552640 9761 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:28.662504 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:28.662672 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:28.762916 9774 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:28.763123 9774 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:28.763818 9774 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:28.764624 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:58:28.764817 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:58:28.764915 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:28.765167 9774 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:30.356397 9795 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:30.356557 9795 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:30.357160 9795 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:30.358000 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:58:30.358199 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:58:30.358311 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:30.358569 9795 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:30.954341 9803 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:58:30.954613 9803 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:58:30.955271 9803 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:58:30.956028 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:58:30.957355 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:58:30.957584 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:58:30.957731 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:58:30.957952 9803 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251028 08:58:32.067817 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 136: this message will repeat every 5th retry.
I20251028 08:58:33.665412 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:33.665570 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:35.210870 9849 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:35.211007 9849 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:35.211660 9849 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:35.212407 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:58:35.212616 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:58:35.212738 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:35.212976 9849 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:35.892262 9857 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:58:35.892467 9857 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:58:35.893028 9857 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:58:35.894886 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:58:35.895030 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
I20251028 08:58:35.895164 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:58:35.895362 9857 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:58:36.832353 9870 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:36.832520 9870 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:58:36.833029 9870 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:58:36.833845 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:58:36.834050 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:58:36.834170 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:36.834408 9870 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:38.667922 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:38.668131 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:38.868343 9890 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:38.868503 9890 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:38.869091 9890 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:38.869838 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:58:38.870016 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:58:38.870126 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:38.870357 9890 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251028 08:58:41.301085 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 141: this message will repeat every 5th retry.
I20251028 08:58:43.670579 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:43.670717 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:58:44.963503 5616 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (540 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:58:44.963527 5887 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (540 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:58:45.574221 5753 heartbeater.cc:646] Failed to heartbeat to 127.4.217.61:33257 (682 consecutive failures): Network error: Failed to ping master at 127.4.217.61:33257: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:58:47.415890 9976 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:47.416057 9976 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:47.416692 9976 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:47.417572 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:58:47.417800 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:58:47.417877 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:47.418113 9976 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:47.419472 9976 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:47.419581 9976 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:47.419973 9976 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:47.420550 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:58:47.420768 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:58:47.420862 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:47.421083 9976 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:48.412099 9976 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:48.412252 9976 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:48.412794 9976 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:48.413545 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:58:48.413800 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:58:48.413883 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:48.414116 9976 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:48.673177 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:48.673348 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:58:51.466507 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 146: this message will repeat every 5th retry.
I20251028 08:58:52.916472 10031 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:52.916676 10031 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:58:52.917315 10031 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:58:52.918030 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:58:52.918282 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:58:52.918382 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:52.918642 10031 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:53.063647 10031 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:58:53.063810 10031 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:58:53.064332 10031 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:58:53.065132 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:58:53.065418 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:58:53.065503 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:58:53.065779 10031 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:58:53.675719 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:53.675881 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:58:54.083207 10045 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:58:54.083371 10045 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:58:54.084136 10045 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:58:54.084617 6761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251028 08:58:54.085829 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:58:54.085956 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:58:54.086064 6759 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:58:54.086298 10045 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:58:58.678382 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:58:58.678529 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251028 08:59:00.976208 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 151: this message will repeat every 5th retry.
I20251028 08:59:02.523336 10127 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:02.523499 10127 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:59:02.524142 10127 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:59:02.524906 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:59:02.525079 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:59:02.525163 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:02.525368 10127 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:03.097494 10127 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:03.097687 10127 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:59:03.098199 10127 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:59:03.098950 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:59:03.099133 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:59:03.099215 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:03.099455 10127 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:03.680910 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:59:03.681062 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:59:06.222383 10160 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:06.222572 10160 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:59:06.223088 10160 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:59:06.223964 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:59:06.224149 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:59:06.224299 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:06.224535 10160 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:07.054181 10179 raft_consensus.cc:493] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:07.054376 10179 raft_consensus.cc:515] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:59:07.054912 10179 leader_election.cc:290] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:59:07.055698 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
W20251028 08:59:07.055908 5773 leader_election.cc:343] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 7a5654d71877406f8a5a4e1e9bb08fa4
I20251028 08:59:07.056001 5773 leader_election.cc:304] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:07.056237 10179 raft_consensus.cc:2749] T 7a5654d71877406f8a5a4e1e9bb08fa4 P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:07.089849 10182 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:59:07.090003 10182 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:59:07.090601 10182 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:59:07.091398 6759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251028 08:59:07.092497 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:59:07.092707 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:59:07.092808 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:59:07.093044 10182 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251028 08:59:08.366753 10194 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:08.366905 10194 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:59:08.367753 10194 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:59:08.368448 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:59:08.368646 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:59:08.368784 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:08.369046 10194 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:08.683284 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:59:08.683452 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:59:08.908749 10205 raft_consensus.cc:493] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:08.908893 10205 raft_consensus.cc:515] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } } peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } }
I20251028 08:59:08.909405 10205 leader_election.cc:290] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085), 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819)
W20251028 08:59:08.910135 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
W20251028 08:59:08.910377 5773 leader_election.cc:343] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: 774e1d1fe9334a45a7df75cc5cd098ff
I20251028 08:59:08.910454 5773 leader_election.cc:304] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:08.910674 10205 raft_consensus.cc:2749] T 774e1d1fe9334a45a7df75cc5cd098ff P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:09.362979 10205 raft_consensus.cc:493] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:09.363138 10205 raft_consensus.cc:515] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:59:09.363729 10205 leader_election.cc:290] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:59:09.364364 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
W20251028 08:59:09.364562 5773 leader_election.cc:343] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: e591ce42e8ce4df0952cb231526544ab
I20251028 08:59:09.364668 5773 leader_election.cc:304] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:09.364878 10205 raft_consensus.cc:2749] T e591ce42e8ce4df0952cb231526544ab P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:10.695909 10221 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8b7d4120e0c84998858b0bd4ed9572b7)
I20251028 08:59:10.696084 10221 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9421be172b482f9c69d4dc5d2b19a8" member_type: VOTER last_known_addr { host: "127.4.217.62" port: 41357 } } peers { permanent_uuid: "8b7d4120e0c84998858b0bd4ed9572b7" member_type: VOTER last_known_addr { host: "127.4.217.61" port: 33257 } } peers { permanent_uuid: "82b026d06b454d65b27216e3002f29af" member_type: VOTER last_known_addr { host: "127.4.217.60" port: 37319 } }
I20251028 08:59:10.696686 10221 leader_election.cc:290] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257), 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319)
W20251028 08:59:10.698599 6759 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 82b026d06b454d65b27216e3002f29af (127.4.217.60:37319): Network error: Client connection negotiation failed: client connection to 127.4.217.60:37319: connect: Connection refused (error 111)
W20251028 08:59:10.698781 6761 leader_election.cc:336] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8b7d4120e0c84998858b0bd4ed9572b7 (127.4.217.61:33257): Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
I20251028 08:59:10.698879 6761 leader_election.cc:304] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9421be172b482f9c69d4dc5d2b19a8; no voters: 82b026d06b454d65b27216e3002f29af, 8b7d4120e0c84998858b0bd4ed9572b7
I20251028 08:59:10.699110 10221 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0b9421be172b482f9c69d4dc5d2b19a8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251028 08:59:10.861690 5640 consensus_peers.cc:597] T 7ff55e6d288f44428891d6fc7e93e2d2 P 52a17de1c37d4a339d1fcb35b550aad7 -> Peer bbab3d08643e49e786c2174faaa741d9 (127.4.217.3:33455): Couldn't send request to peer bbab3d08643e49e786c2174faaa741d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff55e6d288f44428891d6fc7e93e2d2. This is attempt 156: this message will repeat every 5th retry.
I20251028 08:59:12.977741 10244 raft_consensus.cc:493] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 08:59:12.977932 10244 raft_consensus.cc:515] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a17de1c37d4a339d1fcb35b550aad7" member_type: VOTER last_known_addr { host: "127.4.217.2" port: 35819 } } peers { permanent_uuid: "418c8c132d8141ae96893d82e0217aed" member_type: VOTER last_known_addr { host: "127.4.217.1" port: 42085 } } peers { permanent_uuid: "bbab3d08643e49e786c2174faaa741d9" member_type: VOTER last_known_addr { host: "127.4.217.3" port: 33455 } }
I20251028 08:59:12.978538 10244 leader_election.cc:290] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819), 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085)
W20251028 08:59:12.979200 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52a17de1c37d4a339d1fcb35b550aad7 (127.4.217.2:35819): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
W20251028 08:59:12.979385 5773 leader_election.cc:343] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 418c8c132d8141ae96893d82e0217aed (127.4.217.1:42085): Not found: Tablet not found: f0b0ea645082479fa72ede160effe9ea
I20251028 08:59:12.979453 5773 leader_election.cc:304] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbab3d08643e49e786c2174faaa741d9; no voters: 418c8c132d8141ae96893d82e0217aed, 52a17de1c37d4a339d1fcb35b550aad7
I20251028 08:59:12.979655 10244 raft_consensus.cc:2749] T f0b0ea645082479fa72ede160effe9ea P bbab3d08643e49e786c2174faaa741d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251028 08:59:13.685454 6738 master_runner.cc:186] Error getting master registration for 127.4.217.61:33257: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111),
I20251028 08:59:13.685634 6738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251028 08:59:14.253633 6764 maintenance_manager.cc:419] P 0b9421be172b482f9c69d4dc5d2b19a8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1543 bytes on disk
I20251028 08:59:14.258345 6763 maintenance_manager.cc:643] P 0b9421be172b482f9c69d4dc5d2b19a8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":379,"thread_start_us":454,"threads_started":2,"wal-append.queue_time_us":277}
W20251028 08:59:14.447089 5895 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:59:14.447383 5895 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:59:14.447501 5895 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
W20251028 08:59:14.447644 5895 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
F20251028 08:59:14.459667 5915 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1761641954 (unix time) try "date -d @1761641954" if you are using GNU date ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465446 5902 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d9d93bde7fa47158f8e7b574783370e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800001364) received by PID 4964 (TID 0x7f258f790700) from PID 4964; stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465446 5902 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d9d93bde7fa47158f8e7b574783370e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.466650 5918 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465446 5902 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d9d93bde7fa47158f8e7b574783370e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.466650 5918 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467811 5921 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465446 5902 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d9d93bde7fa47158f8e7b574783370e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.466650 5918 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467811 5921 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467959 5910 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c496c6ebeb1d44248b341428961d8939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f25a9352980 (unknown) at ??:0
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465446 5902 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d9d93bde7fa47158f8e7b574783370e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.466650 5918 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467811 5921 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467959 5910 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c496c6ebeb1d44248b341428961d8939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.468327 5911 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465446 5902 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d9d93bde7fa47158f8e7b574783370e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.466650 5918 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467811 5921 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467959 5910 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c496c6ebeb1d44248b341428961d8939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.468327 5911 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.468458 5906 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b01e4320f600411a872c6485f421dd5c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251028 08:59:14.460680 5922 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.460832 5913 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461215 5907 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_541b7acce93a4f9785758f2d9acc4406 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461252 5909 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe01d9e3806c4d75ba49e8aaa6771b3b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461372 5903 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c22aee04efca4315947c85505204085f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461527 5927 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461826 5917 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.461972 5912 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462566 5923 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462735 5905 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_80a36344c9054a9e996f831074abfc41 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.462801 5908 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5ab4d2da1704bf4a0c944f5c59dc3f7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464063 5925 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464128 5901 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3fd0e038ef9e4c7f8ae3f98b9a95181b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.464249 5914 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465440 5904 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c896dce32fc04719beb2467b2bf1f855 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.465446 5902 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d9d93bde7fa47158f8e7b574783370e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.466650 5918 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467811 5921 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.467959 5910 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c496c6ebeb1d44248b341428961d8939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.468327 5911 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.468458 5906 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b01e4320f600411a872c6485f421dd5c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)F20251028 08:59:14.468727 5916 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.217.62:41357,127.4.217.61:33257,127.4.217.60:37319, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.217.61:33257: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f25a2128fb7 gsignal at ??:0
@ 0x7f25a212a921 abort at ??:0
@ 0x7f25a4e60dcd google::LogMessage::Fail() at ??:0
@ 0x7f25a4e64b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f25a4e607cc google::LogMessage::Flush() at ??:0
@ 0x7f25a4e61f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40c481 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:328
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f25a27b06df (unknown) at ??:0
@ 0x7f25a93476db start_thread at ??:0
@ 0x7f25a220b71f clone at ??:0