Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250830 01:58:02.433923 27047 test_util.cc:276] Using random seed: 2012421460
2025-08-30T01:58:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-30T01:58:02Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskqeOeTE/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskqeOeTE/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/security.properties
2025-08-30 01:58:08: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskqeOeTE/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskqeOeTE/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250830 01:58:18.815163 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--webserver_interface=127.26.105.254
--webserver_port=0
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:19.151369 27290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:19.151733 27290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:19.151860 27290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:19.160171 27290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:19.160282 27290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:19.160367 27290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:19.160432 27290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:19.170295 27290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.105.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:19.172333 27290 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:19.175347 27290 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:19.184146 27296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:19.184146 27295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:19.184573 27298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:19.186375 27290 server_base.cc:1047] running on GCE node
I20250830 01:58:19.187486 27290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:19.189065 27290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:19.190312 27290 hybrid_clock.cc:648] HybridClock initialized: now 1756519099190259 us; error 42 us; skew 500 ppm
I20250830 01:58:19.190701 27290 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:19.193125 27290 webserver.cc:480] Webserver started at http://127.26.105.254:37963/ using document root <none> and password file <none>
I20250830 01:58:19.193912 27290 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:19.194023 27290 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:19.194536 27290 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250830 01:58:19.197502 27290 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/instance:
uuid: "ad9824f7e6604077ac8a034879b4c731"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:19.198398 27290 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal/instance:
uuid: "ad9824f7e6604077ac8a034879b4c731"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:19.204319 27290 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20250830 01:58:19.207998 27304 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:19.209394 27290 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250830 01:58:19.209576 27290 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
uuid: "ad9824f7e6604077ac8a034879b4c731"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:19.209800 27290 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:19.248960 27290 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:19.249912 27290 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:19.250211 27290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:19.270709 27290 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.254:36635
I20250830 01:58:19.270766 27355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.254:36635 every 8 connection(s)
I20250830 01:58:19.272238 27290 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
I20250830 01:58:19.274168 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 27290
I20250830 01:58:19.274485 27047 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal/instance
I20250830 01:58:19.276305 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:19.277238 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.253:39521
--webserver_interface=127.26.105.253
--webserver_port=0
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250830 01:58:19.282436 27356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
I20250830 01:58:19.304610 27356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } has no permanent_uuid. Determining permanent_uuid...
W20250830 01:58:19.305824 27307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:19.307418 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:19.356771 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } attempt: 1
W20250830 01:58:19.358795 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:19.427045 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } attempt: 2
W20250830 01:58:19.429188 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:19.520576 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } attempt: 3
W20250830 01:58:19.523010 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:19.617012 27357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:19.617630 27357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:19.618062 27357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:19.629456 27357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:19.629575 27357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:19.629670 27357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:19.629740 27357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:19.641216 27357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.253:39521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.105.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:19.642896 27357 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:19.644629 27357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:19.652491 27365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:19.652592 27366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:19.652915 27368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:19.653828 27357 server_base.cc:1047] running on GCE node
I20250830 01:58:19.654625 27357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:19.656093 27357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:19.657310 27357 hybrid_clock.cc:648] HybridClock initialized: now 1756519099657251 us; error 53 us; skew 500 ppm
I20250830 01:58:19.657660 27357 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:19.659906 27357 webserver.cc:480] Webserver started at http://127.26.105.253:37387/ using document root <none> and password file <none>
I20250830 01:58:19.660640 27357 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:19.660730 27357 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:19.661070 27357 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250830 01:58:19.663496 27357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/instance:
uuid: "9ba697035f304124a88a1cf1aabb748e"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:19.664276 27357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal/instance:
uuid: "9ba697035f304124a88a1cf1aabb748e"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:19.666292 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } attempt: 4
W20250830 01:58:19.668030 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:19.669545 27357 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20250830 01:58:19.672971 27374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:19.674396 27357 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250830 01:58:19.674556 27357 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
uuid: "9ba697035f304124a88a1cf1aabb748e"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:19.674734 27357 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:19.692723 27357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:19.693583 27357 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:19.693835 27357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:19.715001 27357 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.253:39521
I20250830 01:58:19.714951 27425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.253:39521 every 8 connection(s)
I20250830 01:58:19.716549 27357 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
I20250830 01:58:19.720394 27426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:19.725313 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 27357
I20250830 01:58:19.725459 27047 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal/instance
I20250830 01:58:19.726606 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--webserver_interface=127.26.105.252
--webserver_port=0
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250830 01:58:19.726274 27426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
I20250830 01:58:19.739269 27426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } has no permanent_uuid. Determining permanent_uuid...
I20250830 01:58:19.750419 27426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
W20250830 01:58:19.751510 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:19.753114 27426 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:19.802429 27426 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 1
W20250830 01:58:19.804337 27426 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:19.872627 27426 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 2
W20250830 01:58:19.874519 27426 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:19.965801 27426 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 3
I20250830 01:58:19.967276 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } attempt: 5
W20250830 01:58:19.967857 27426 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:19.973536 27356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
W20250830 01:58:19.975124 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:20.026379 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 1
W20250830 01:58:20.028546 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:20.053036 27428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:20.053357 27428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:20.053489 27428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:20.061750 27428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:20.061854 27428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:20.061924 27428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:20.061995 27428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:20.071918 27428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.105.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:20.073562 27428 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:20.075357 27428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:20.084321 27435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:20.084388 27436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:20.084523 27438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:20.085860 27428 server_base.cc:1047] running on GCE node
I20250830 01:58:20.086920 27428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:20.088438 27428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:20.089658 27428 hybrid_clock.cc:648] HybridClock initialized: now 1756519100089596 us; error 56 us; skew 500 ppm
I20250830 01:58:20.089984 27428 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:20.092209 27428 webserver.cc:480] Webserver started at http://127.26.105.252:39345/ using document root <none> and password file <none>
I20250830 01:58:20.092926 27428 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:20.093043 27428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:20.093375 27428 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250830 01:58:20.095801 27428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/instance:
uuid: "fd4f5c1e7015460e99715d8a2edd2959"
format_stamp: "Formatted at 2025-08-30 01:58:20 on dist-test-slave-gp2r"
I20250830 01:58:20.096597 27428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal/instance:
uuid: "fd4f5c1e7015460e99715d8a2edd2959"
format_stamp: "Formatted at 2025-08-30 01:58:20 on dist-test-slave-gp2r"
I20250830 01:58:20.096819 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 2
W20250830 01:58:20.098997 27356 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:20.102715 27428 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250830 01:58:20.106269 27444 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:20.108449 27428 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250830 01:58:20.108726 27428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
uuid: "fd4f5c1e7015460e99715d8a2edd2959"
format_stamp: "Formatted at 2025-08-30 01:58:20 on dist-test-slave-gp2r"
I20250830 01:58:20.109102 27428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:20.111130 27426 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 4
W20250830 01:58:20.112847 27426 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:20.126344 27428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:20.127206 27428 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:20.127447 27428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:20.147197 27428 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.252:46233
I20250830 01:58:20.147221 27495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.252:46233 every 8 connection(s)
I20250830 01:58:20.148654 27428 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
I20250830 01:58:20.152431 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:20.153589 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 27428
I20250830 01:58:20.153760 27047 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal/instance
I20250830 01:58:20.158357 27496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
I20250830 01:58:20.174013 27496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } has no permanent_uuid. Determining permanent_uuid...
I20250830 01:58:20.182909 27496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20250830 01:58:20.197949 27496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap starting.
I20250830 01:58:20.201061 27496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:20.202502 27496 log.cc:826] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:20.204895 27496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: No bootstrap required, opened a new log
I20250830 01:58:20.205226 27356 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 3
I20250830 01:58:20.210551 27496 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.211539 27496 raft_consensus.cc:383] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:20.211676 27496 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4f5c1e7015460e99715d8a2edd2959, State: Initialized, Role: FOLLOWER
I20250830 01:58:20.212378 27496 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.213372 27504 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.213666 27504 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:20.214555 27496 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:20.215451 27356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap starting.
I20250830 01:58:20.219221 27356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:20.220961 27356 log.cc:826] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:20.224529 27356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: No bootstrap required, opened a new log
I20250830 01:58:20.231169 27356 raft_consensus.cc:357] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.231639 27356 raft_consensus.cc:383] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:20.231796 27356 raft_consensus.cc:738] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Initialized, Role: FOLLOWER
I20250830 01:58:20.233443 27356 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
W20250830 01:58:20.234872 27519 catalog_manager.cc:1560] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250830 01:58:20.235064 27519 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250830 01:58:20.239359 27522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.240442 27522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:20.234537 27428 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: OK, instance_id { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" instance_seqno: 1756519099265104 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250830 01:58:20.241581 27428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:20.241703 27356 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: configured and running, proceeding with master startup.
W20250830 01:58:20.252964 27310 tablet.cc:2378] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250830 01:58:20.281213 27537 catalog_manager.cc:1560] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250830 01:58:20.281473 27537 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250830 01:58:20.267208 27290 master_runner.cc:186] Error getting master registration for 127.26.105.253:39521: OK, instance_id { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" instance_seqno: 1756519099708950 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250830 01:58:20.293736 27290 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:58:20.379541 27450 tablet.cc:2378] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250830 01:58:20.412109 27426 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } attempt: 5
I20250830 01:58:20.423826 27426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Bootstrap starting.
I20250830 01:58:20.427024 27426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:20.428094 27426 log.cc:826] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:20.430655 27426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: No bootstrap required, opened a new log
I20250830 01:58:20.435528 27426 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.436225 27426 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:20.436352 27426 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba697035f304124a88a1cf1aabb748e, State: Initialized, Role: FOLLOWER
I20250830 01:58:20.438166 27426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.439236 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.439585 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:20.440383 27426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: configured and running, proceeding with master startup.
W20250830 01:58:20.446373 27380 tablet.cc:2378] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250830 01:58:20.463711 27504 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:20.463985 27504 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.465669 27504 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
I20250830 01:58:20.466785 27401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e" is_pre_election: true
I20250830 01:58:20.467335 27401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd4f5c1e7015460e99715d8a2edd2959 in term 0.
I20250830 01:58:20.467655 27555 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250830 01:58:20.468237 27447 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959; no voters:
I20250830 01:58:20.469005 27504 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:20.469156 27504 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:20.469242 27504 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:20.469436 27331 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:20.469949 27331 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd4f5c1e7015460e99715d8a2edd2959 in term 0.
I20250830 01:58:20.477906 27504 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.482506 27357 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20250830 01:58:20.482731 27357 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:20.483734 27331 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731"
I20250830 01:58:20.483959 27331 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:20.489356 27504 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 1 election: Requested vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
I20250830 01:58:20.489735 27401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e"
I20250830 01:58:20.489956 27401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:20.491344 27331 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd4f5c1e7015460e99715d8a2edd2959 in term 1.
I20250830 01:58:20.493399 27448 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959; no voters:
I20250830 01:58:20.493834 27504 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:20.493841 27401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd4f5c1e7015460e99715d8a2edd2959 in term 1.
I20250830 01:58:20.494699 27504 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 1 LEADER]: Becoming Leader. State: Replica: fd4f5c1e7015460e99715d8a2edd2959, State: Running, Role: LEADER
I20250830 01:58:20.495321 27504 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:20.498878 27556 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd4f5c1e7015460e99715d8a2edd2959. Latest consensus state: current_term: 1 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.499240 27556 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:20.504629 27558 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:20.508190 27558 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:20.521939 27401 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Refusing update from remote peer fd4f5c1e7015460e99715d8a2edd2959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250830 01:58:20.523473 27331 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Refusing update from remote peer fd4f5c1e7015460e99715d8a2edd2959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250830 01:58:20.524096 27556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:20.524370 27504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:20.546509 27522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd4f5c1e7015460e99715d8a2edd2959. Latest consensus state: current_term: 1 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.546795 27522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:20.551232 27559 mvcc.cc:204] Tried to move back new op lower bound from 7194702235712004096 to 7194702235633586176. Current Snapshot: MvccSnapshot[applied={T|T < 7194702235712004096}]
I20250830 01:58:20.554198 27561 mvcc.cc:204] Tried to move back new op lower bound from 7194702235712004096 to 7194702235633586176. Current Snapshot: MvccSnapshot[applied={T|T < 7194702235712004096}]
I20250830 01:58:20.554838 27504 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.555255 27504 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:20.558547 27558 catalog_manager.cc:1349] Generated new cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:20.558655 27558 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:20.560437 27504 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.560676 27504 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:20.571794 27522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.572053 27522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:20.576931 27558 catalog_manager.cc:1372] Generated new certificate authority record
I20250830 01:58:20.578755 27558 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:20.591271 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd4f5c1e7015460e99715d8a2edd2959. Latest consensus state: current_term: 1 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.591569 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:20.599716 27566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:20.600178 27566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:20.600419 27558 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Generated new TSK 0
I20250830 01:58:20.601550 27558 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:20.602952 27558 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:20.604521 27558 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250830 01:58:20.627691 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.105.193:0
--local_ip_for_outbound_sockets=127.26.105.193
--webserver_interface=127.26.105.193
--webserver_port=0
--tserver_master_addrs=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250830 01:58:20.986752 27571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:20.987082 27571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:20.987228 27571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:20.995882 27571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:20.996358 27571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.105.193
I20250830 01:58:21.007171 27571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.105.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.105.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.105.193
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:21.008823 27571 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:21.010648 27571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:21.020838 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:21.020771 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:21.022308 27579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:21.022965 27571 server_base.cc:1047] running on GCE node
I20250830 01:58:21.023577 27571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:21.024572 27571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:21.025815 27571 hybrid_clock.cc:648] HybridClock initialized: now 1756519101025762 us; error 44 us; skew 500 ppm
I20250830 01:58:21.026207 27571 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:21.028508 27571 webserver.cc:480] Webserver started at http://127.26.105.193:39919/ using document root <none> and password file <none>
I20250830 01:58:21.029279 27571 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:21.029388 27571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:21.029737 27571 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250830 01:58:21.032259 27571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data/instance:
uuid: "03d61552c37a44ad9a7a6f6f5c236227"
format_stamp: "Formatted at 2025-08-30 01:58:21 on dist-test-slave-gp2r"
I20250830 01:58:21.033047 27571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal/instance:
uuid: "03d61552c37a44ad9a7a6f6f5c236227"
format_stamp: "Formatted at 2025-08-30 01:58:21 on dist-test-slave-gp2r"
I20250830 01:58:21.038515 27571 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250830 01:58:21.042251 27585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:21.044168 27571 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250830 01:58:21.044359 27571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal
uuid: "03d61552c37a44ad9a7a6f6f5c236227"
format_stamp: "Formatted at 2025-08-30 01:58:21 on dist-test-slave-gp2r"
I20250830 01:58:21.044574 27571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:21.060649 27571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:21.061532 27571 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:21.061811 27571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:21.062851 27571 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250830 01:58:21.064577 27571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250830 01:58:21.064684 27571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:21.064798 27571 ts_tablet_manager.cc:610] Registered 0 tablets
I20250830 01:58:21.064873 27571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:21.101783 27678 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fd4f5c1e7015460e99715d8a2edd2959)
I20250830 01:58:21.102080 27678 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:21.103930 27678 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:21.104554 27400 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e" is_pre_election: true
I20250830 01:58:21.104737 27471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd4f5c1e7015460e99715d8a2edd2959" is_pre_election: true
I20250830 01:58:21.105784 27307 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:21.106369 27678 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250830 01:58:21.109755 27571 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.193:36219
I20250830 01:58:21.109819 27698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.193:36219 every 8 connection(s)
I20250830 01:58:21.111189 27571 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/data/info.pb
I20250830 01:58:21.121973 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 27571
I20250830 01:58:21.122298 27047 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal/instance
I20250830 01:58:21.128212 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.105.194:0
--local_ip_for_outbound_sockets=127.26.105.194
--webserver_interface=127.26.105.194
--webserver_port=0
--tserver_master_addrs=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250830 01:58:21.136775 27701 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:21.136844 27700 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:21.137266 27701 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:21.137269 27700 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:21.137382 27699 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:21.137501 27699 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:21.138696 27701 heartbeater.cc:507] Master 127.26.105.253:39521 requested a full tablet report, sending...
I20250830 01:58:21.138747 27699 heartbeater.cc:507] Master 127.26.105.252:46233 requested a full tablet report, sending...
I20250830 01:58:21.138792 27700 heartbeater.cc:507] Master 127.26.105.254:36635 requested a full tablet report, sending...
I20250830 01:58:21.141070 27391 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:21.141191 27461 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:21.141232 27321 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:21.143113 27461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.105.193:57039
I20250830 01:58:21.285254 27537 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:21.285373 27537 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: loading cluster ID for follower catalog manager: success
I20250830 01:58:21.289767 27537 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: acquiring CA information for follower catalog manager: success
I20250830 01:58:21.292281 27537 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:21.473701 27555 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:21.473847 27555 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: loading cluster ID for follower catalog manager: success
I20250830 01:58:21.478158 27555 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: acquiring CA information for follower catalog manager: success
I20250830 01:58:21.480788 27555 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:21.481284 27711 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fd4f5c1e7015460e99715d8a2edd2959)
I20250830 01:58:21.481496 27711 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:21.483081 27711 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:21.483677 27471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd4f5c1e7015460e99715d8a2edd2959" is_pre_election: true
I20250830 01:58:21.483958 27331 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:21.484789 27377 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:21.485212 27711 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250830 01:58:21.504150 27709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:21.504464 27709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:21.504597 27709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:21.512944 27709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:21.513151 27709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.105.194
I20250830 01:58:21.523765 27709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.105.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.105.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.105.194
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:21.525362 27709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:21.527135 27709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:21.537896 27716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:21.539163 27717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:21.539945 27719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:21.545935 27709 server_base.cc:1047] running on GCE node
I20250830 01:58:21.546627 27709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:21.547613 27709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:21.548837 27709 hybrid_clock.cc:648] HybridClock initialized: now 1756519101548776 us; error 57 us; skew 500 ppm
I20250830 01:58:21.549163 27709 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:21.552019 27709 webserver.cc:480] Webserver started at http://127.26.105.194:43587/ using document root <none> and password file <none>
I20250830 01:58:21.552796 27709 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:21.552911 27709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:21.553263 27709 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250830 01:58:21.555701 27709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data/instance:
uuid: "7f5563a27e4e4748813d624aab13cdca"
format_stamp: "Formatted at 2025-08-30 01:58:21 on dist-test-slave-gp2r"
I20250830 01:58:21.556444 27709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal/instance:
uuid: "7f5563a27e4e4748813d624aab13cdca"
format_stamp: "Formatted at 2025-08-30 01:58:21 on dist-test-slave-gp2r"
I20250830 01:58:21.562012 27709 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250830 01:58:21.565856 27725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:21.567395 27709 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250830 01:58:21.567565 27709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal
uuid: "7f5563a27e4e4748813d624aab13cdca"
format_stamp: "Formatted at 2025-08-30 01:58:21 on dist-test-slave-gp2r"
I20250830 01:58:21.567759 27709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:21.593014 27709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:21.594074 27709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:21.594390 27709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:21.595439 27709 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250830 01:58:21.597105 27709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250830 01:58:21.597203 27709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:21.597328 27709 ts_tablet_manager.cc:610] Registered 0 tablets
I20250830 01:58:21.597409 27709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:21.639676 27709 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.194:43787
I20250830 01:58:21.639716 27837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.194:43787 every 8 connection(s)
I20250830 01:58:21.641510 27709 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/data/info.pb
I20250830 01:58:21.643368 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 27709
I20250830 01:58:21.643566 27047 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal/instance
I20250830 01:58:21.645591 27699 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
I20250830 01:58:21.650508 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.105.195:0
--local_ip_for_outbound_sockets=127.26.105.195
--webserver_interface=127.26.105.195
--webserver_port=0
--tserver_master_addrs=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250830 01:58:21.686826 27838 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:21.687347 27838 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:21.688623 27838 heartbeater.cc:507] Master 127.26.105.252:46233 requested a full tablet report, sending...
I20250830 01:58:21.690831 27461 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:21.692229 27461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.105.194:37105
I20250830 01:58:21.697057 27839 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:21.697288 27839 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:21.699448 27839 heartbeater.cc:507] Master 127.26.105.254:36635 requested a full tablet report, sending...
I20250830 01:58:21.707616 27320 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:21.711978 27842 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:21.712154 27842 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:21.712471 27842 heartbeater.cc:507] Master 127.26.105.253:39521 requested a full tablet report, sending...
I20250830 01:58:21.713425 27391 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
W20250830 01:58:22.043834 27845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:22.044166 27845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:22.044261 27845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:22.052879 27845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:22.053069 27845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.105.195
I20250830 01:58:22.063706 27845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.105.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.105.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.105.195
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:22.065303 27845 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:22.067181 27845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:22.077282 27852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:22.077534 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:22.078521 27855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:22.079691 27845 server_base.cc:1047] running on GCE node
I20250830 01:58:22.080351 27845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:22.081362 27845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:22.082568 27845 hybrid_clock.cc:648] HybridClock initialized: now 1756519102082521 us; error 37 us; skew 500 ppm
I20250830 01:58:22.082866 27845 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:22.085731 27845 webserver.cc:480] Webserver started at http://127.26.105.195:35251/ using document root <none> and password file <none>
I20250830 01:58:22.086566 27845 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:22.086668 27845 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:22.086988 27845 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250830 01:58:22.089489 27845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data/instance:
uuid: "893b0b641a51432f9172ed0b41521822"
format_stamp: "Formatted at 2025-08-30 01:58:22 on dist-test-slave-gp2r"
I20250830 01:58:22.090296 27845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal/instance:
uuid: "893b0b641a51432f9172ed0b41521822"
format_stamp: "Formatted at 2025-08-30 01:58:22 on dist-test-slave-gp2r"
I20250830 01:58:22.096395 27845 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20250830 01:58:22.100988 27861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:22.102703 27845 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250830 01:58:22.102888 27845 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal
uuid: "893b0b641a51432f9172ed0b41521822"
format_stamp: "Formatted at 2025-08-30 01:58:22 on dist-test-slave-gp2r"
I20250830 01:58:22.103096 27845 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:22.136690 27845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:22.137579 27845 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:22.137848 27845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:22.138860 27845 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250830 01:58:22.140612 27845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250830 01:58:22.140705 27845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:22.140807 27845 ts_tablet_manager.cc:610] Registered 0 tablets
I20250830 01:58:22.140877 27845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:22.187666 27845 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.195:35297
I20250830 01:58:22.187736 27973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.195:35297 every 8 connection(s)
I20250830 01:58:22.189226 27845 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/data/info.pb
I20250830 01:58:22.190858 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 27845
I20250830 01:58:22.191071 27047 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal/instance
I20250830 01:58:22.196234 27838 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
I20250830 01:58:22.214427 27975 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:22.214874 27975 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:22.215901 27976 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:22.216048 27976 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:22.216096 27975 heartbeater.cc:507] Master 127.26.105.254:36635 requested a full tablet report, sending...
I20250830 01:58:22.216591 27976 heartbeater.cc:507] Master 127.26.105.253:39521 requested a full tablet report, sending...
I20250830 01:58:22.217852 27391 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:22.217905 27320 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:22.218771 27974 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:22.218896 27974 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:22.219383 27974 heartbeater.cc:507] Master 127.26.105.252:46233 requested a full tablet report, sending...
I20250830 01:58:22.220219 27460 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:22.221694 27460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.105.195:38951
I20250830 01:58:22.233469 27047 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250830 01:58:22.267228 27460 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_badff657ae07402a88632f78ec1befb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.267304 27459 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_739ceee58b144d72b56342855499ecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.267333 27461 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_2c4fc026b80045d8a58497a870104278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.268090 27458 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_624ff0828e534d54a8ea5f69551f7e8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.268586 27456 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_cef396d7e593458aa7679269acecf621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.268586 27457 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_a36b1b6bfc854dfc907635b41d47be02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.271279 27455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_2c93d305666e48d5b919f93ebe9e05ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.271587 27454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_488d167c0efe4d1b91a27f563856f9f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.272886 27453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_d19bb3dab12e478cbd8897446f84d508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.273260 27452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "default.Table_11641b1a96984fa099ffa232b6a0033e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250830 01:58:22.278729 27460 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_badff657ae07402a88632f78ec1befb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250830 01:58:22.295511 27047 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu pid 27428
I20250830 01:58:22.297029 27491 generic_service.cc:196] Checking for leaks (request via RPC)
I20250830 01:58:22.426987 27047 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu with pid 27428
W20250830 01:58:22.455554 27984 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.105.252:46233)
I20250830 01:58:22.484848 28022 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fd4f5c1e7015460e99715d8a2edd2959)
I20250830 01:58:22.485492 28022 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:22.486932 27331 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:22.500481 28022 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:22.501008 27377 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:22.501284 27377 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:22.501735 28022 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250830 01:58:22.509857 28025 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fd4f5c1e7015460e99715d8a2edd2959)
I20250830 01:58:22.510268 28025 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:22.511590 27400 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e" is_pre_election: true
I20250830 01:58:22.511837 27400 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 1.
I20250830 01:58:22.518362 28025 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:22.522866 27307 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters:
I20250830 01:58:22.523221 28025 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250830 01:58:22.523424 28025 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fd4f5c1e7015460e99715d8a2edd2959)
I20250830 01:58:22.523631 28025 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 1 FOLLOWER]: Advancing to term 2
I20250830 01:58:22.528842 28025 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:22.529968 28025 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 election: Requested vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:22.533067 27400 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e"
I20250830 01:58:22.533327 27400 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 1 FOLLOWER]: Advancing to term 2
W20250830 01:58:22.533360 27307 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:22.534667 27307 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:22.536226 27400 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 2.
I20250830 01:58:22.536695 27307 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters: fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:22.537524 28025 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 FOLLOWER]: Leader election won for term 2
I20250830 01:58:22.540424 28025 raft_consensus.cc:695] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 LEADER]: Becoming Leader. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Running, Role: LEADER
I20250830 01:58:22.541038 28025 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:22.550209 28028 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 2 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:22.550774 28028 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:22.553843 28030 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:22.556188 28030 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:22.557176 28030 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:22.557314 28030 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:22.558869 28030 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:22.559962 28030 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Loaded TSK: 0
I20250830 01:58:22.561311 28030 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:22.562414 28030 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:22.563460 28030 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250830 01:58:22.570739 27321 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_d19bb3dab12e478cbd8897446f84d508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.571810 27319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_488d167c0efe4d1b91a27f563856f9f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.572489 27318 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_2c93d305666e48d5b919f93ebe9e05ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.573108 27317 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_2c4fc026b80045d8a58497a870104278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.573913 27316 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_badff657ae07402a88632f78ec1befb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.575395 27314 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_624ff0828e534d54a8ea5f69551f7e8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.575544 27313 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_a36b1b6bfc854dfc907635b41d47be02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.575704 27315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_739ceee58b144d72b56342855499ecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.576196 27312 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_cef396d7e593458aa7679269acecf621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:22.576412 27320 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "default.Table_11641b1a96984fa099ffa232b6a0033e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250830 01:58:22.584774 27318 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2c93d305666e48d5b919f93ebe9e05ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:22.586647 27319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_488d167c0efe4d1b91a27f563856f9f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:22.588541 27317 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2c4fc026b80045d8a58497a870104278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:22.590109 27321 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d19bb3dab12e478cbd8897446f84d508 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:22.653110 27699 heartbeater.cc:646] Failed to heartbeat to 127.26.105.252:46233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:22.700253 27838 heartbeater.cc:646] Failed to heartbeat to 127.26.105.252:46233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:22.723747 27974 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
W20250830 01:58:22.725440 27974 heartbeater.cc:646] Failed to heartbeat to 127.26.105.252:46233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:22.810813 27319 catalog_manager.cc:2149] created HMS catalog entry for table [id=a66ba6403ba349f3ade7628ab65cd30f]
I20250830 01:58:22.815847 27400 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Refusing update from remote peer ad9824f7e6604077ac8a034879b4c731: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20250830 01:58:22.816378 27307 consensus_peers.cc:489] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 -> Peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Couldn't send request to peer fd4f5c1e7015460e99715d8a2edd2959. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250830 01:58:22.816756 28028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250830 01:58:22.834388 28022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 2 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:22.834650 28022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:22.838042 28035 mvcc.cc:204] Tried to move back new op lower bound from 7194702245123411968 to 7194702244013260800. Current Snapshot: MvccSnapshot[applied={T|T < 7194702245123411968}]
I20250830 01:58:22.842290 28028 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:22.842558 28028 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:22.843071 28022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:22.843498 28022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:22.848755 27319 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "default.Table_488d167c0efe4d1b91a27f563856f9f6" } new_table_name: "default.Table_59d8d3858b8c490f984614c9187eef7c" modify_external_catalogs: true
I20250830 01:58:22.865460 27317 catalog_manager.cc:2149] created HMS catalog entry for table [id=65cc6cabcfd4405994a16c620558df46]
I20250830 01:58:22.883514 27317 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "default.Table_2c4fc026b80045d8a58497a870104278" } new_table_name: "default.Table_f8d128ad097b4e25a4cc03e5b2f02676" modify_external_catalogs: true
I20250830 01:58:22.915596 27909 tablet_service.cc:1468] Processing CreateTablet for tablet fc86731b9fa247619aa9414e3709ba35 (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.917321 27909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc86731b9fa247619aa9414e3709ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.917752 27908 tablet_service.cc:1468] Processing CreateTablet for tablet 177a8293c20c412ea767995c5c9af204 (DEFAULT_TABLE table=default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.918813 27908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 177a8293c20c412ea767995c5c9af204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.919981 27773 tablet_service.cc:1468] Processing CreateTablet for tablet fc86731b9fa247619aa9414e3709ba35 (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.921114 27773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc86731b9fa247619aa9414e3709ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.922806 27772 tablet_service.cc:1468] Processing CreateTablet for tablet 177a8293c20c412ea767995c5c9af204 (DEFAULT_TABLE table=default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.923391 27772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 177a8293c20c412ea767995c5c9af204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.927528 27633 tablet_service.cc:1468] Processing CreateTablet for tablet fc86731b9fa247619aa9414e3709ba35 (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.929420 27633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc86731b9fa247619aa9414e3709ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.934573 27632 tablet_service.cc:1468] Processing CreateTablet for tablet 177a8293c20c412ea767995c5c9af204 (DEFAULT_TABLE table=default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.935245 27632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 177a8293c20c412ea767995c5c9af204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.938366 28046 tablet_bootstrap.cc:492] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:22.939512 28047 tablet_bootstrap.cc:492] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:22.945530 28047 tablet_bootstrap.cc:654] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:22.946928 28047 log.cc:826] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:22.948597 28046 tablet_bootstrap.cc:654] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:22.950017 28046 log.cc:826] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:22.954137 28047 tablet_bootstrap.cc:492] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:22.954474 28047 ts_tablet_manager.cc:1397] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20250830 01:58:22.956108 27321 catalog_manager.cc:2149] created HMS catalog entry for table [id=e9fceb6dca37466384f24c8bde25f71a]
I20250830 01:58:22.963096 28048 tablet_bootstrap.cc:492] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:22.966742 27321 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:54338
I20250830 01:58:22.967146 28048 tablet_bootstrap.cc:654] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:22.968580 28048 log.cc:826] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:22.971081 28046 tablet_bootstrap.cc:492] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:22.971439 28046 ts_tablet_manager.cc:1397] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20250830 01:58:22.975380 27909 tablet_service.cc:1468] Processing CreateTablet for tablet cddb4c87468a44d1a2bae0fbd6784a36 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.975832 27909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddb4c87468a44d1a2bae0fbd6784a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.978981 28046 raft_consensus.cc:357] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:22.979542 28046 raft_consensus.cc:383] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:22.980285 27772 tablet_service.cc:1468] Processing CreateTablet for tablet cddb4c87468a44d1a2bae0fbd6784a36 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.980701 27772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddb4c87468a44d1a2bae0fbd6784a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.985119 27321 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Sending DeleteTablet for 3 replicas of tablet 177a8293c20c412ea767995c5c9af204
I20250830 01:58:22.985879 27321 catalog_manager.cc:6652] ReplaceTablet: tablet 177a8293c20c412ea767995c5c9af204 deleted and replaced by tablet 3996bdda030340d081af363dbdfea328
I20250830 01:58:22.986245 27321 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:54338
I20250830 01:58:22.986568 28048 tablet_bootstrap.cc:492] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:22.986747 27321 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:54338
I20250830 01:58:22.986867 28048 ts_tablet_manager.cc:1397] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20250830 01:58:22.988832 27632 tablet_service.cc:1515] Processing DeleteTablet for tablet 177a8293c20c412ea767995c5c9af204 with delete_type TABLET_DATA_DELETED (replaced by tablet 3996bdda030340d081af363dbdfea328) from {username='slave'} at 127.0.0.1:45222
I20250830 01:58:22.989300 27772 tablet_service.cc:1515] Processing DeleteTablet for tablet 177a8293c20c412ea767995c5c9af204 with delete_type TABLET_DATA_DELETED (replaced by tablet 3996bdda030340d081af363dbdfea328) from {username='slave'} at 127.0.0.1:54160
I20250830 01:58:22.990131 27633 tablet_service.cc:1468] Processing CreateTablet for tablet cddb4c87468a44d1a2bae0fbd6784a36 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:22.992090 27633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddb4c87468a44d1a2bae0fbd6784a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:22.994354 27908 tablet_service.cc:1515] Processing DeleteTablet for tablet 177a8293c20c412ea767995c5c9af204 with delete_type TABLET_DATA_DELETED (replaced by tablet 3996bdda030340d081af363dbdfea328) from {username='slave'} at 127.0.0.1:43544
I20250830 01:58:22.996963 27321 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Sending DeleteTablet for 3 replicas of tablet fc86731b9fa247619aa9414e3709ba35
I20250830 01:58:22.997931 27907 tablet_service.cc:1515] Processing DeleteTablet for tablet fc86731b9fa247619aa9414e3709ba35 with delete_type TABLET_DATA_DELETED (replaced by tablet 47a4f27ca16d436386ed86810020f84b) from {username='slave'} at 127.0.0.1:43544
I20250830 01:58:22.999127 28048 raft_consensus.cc:357] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:22.999548 28048 raft_consensus.cc:383] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:22.999687 28048 raft_consensus.cc:738] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:22.999876 28046 raft_consensus.cc:738] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.001436 27773 tablet_service.cc:1515] Processing DeleteTablet for tablet fc86731b9fa247619aa9414e3709ba35 with delete_type TABLET_DATA_DELETED (replaced by tablet 47a4f27ca16d436386ed86810020f84b) from {username='slave'} at 127.0.0.1:54160
I20250830 01:58:23.002125 28046 consensus_queue.cc:260] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.003814 28048 consensus_queue.cc:260] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.007388 27632 tablet_service.cc:1468] Processing CreateTablet for tablet 3996bdda030340d081af363dbdfea328 (DEFAULT_TABLE table=default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.008409 27907 tablet_service.cc:1468] Processing CreateTablet for tablet 3996bdda030340d081af363dbdfea328 (DEFAULT_TABLE table=default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.009160 27907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3996bdda030340d081af363dbdfea328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:23.014511 27321 catalog_manager.cc:6652] ReplaceTablet: tablet fc86731b9fa247619aa9414e3709ba35 deleted and replaced by tablet 47a4f27ca16d436386ed86810020f84b
I20250830 01:58:23.014786 27321 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:54338
I20250830 01:58:23.015209 27773 tablet_service.cc:1468] Processing CreateTablet for tablet 3996bdda030340d081af363dbdfea328 (DEFAULT_TABLE table=default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.015344 27631 tablet_service.cc:1515] Processing DeleteTablet for tablet fc86731b9fa247619aa9414e3709ba35 with delete_type TABLET_DATA_DELETED (replaced by tablet 47a4f27ca16d436386ed86810020f84b) from {username='slave'} at 127.0.0.1:45222
I20250830 01:58:23.015702 27773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3996bdda030340d081af363dbdfea328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:23.015712 27321 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:54338
I20250830 01:58:23.016216 27321 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "default.Table_d19bb3dab12e478cbd8897446f84d508" } new_table_name: "default.Table_31e5e45019fd41888d6a471fbd3736b4" modify_external_catalogs: true
I20250830 01:58:23.019757 28047 raft_consensus.cc:357] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.020177 28047 raft_consensus.cc:383] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.020316 28047 raft_consensus.cc:738] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.021086 28047 consensus_queue.cc:260] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.026856 27632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3996bdda030340d081af363dbdfea328. 1 dirs total, 0 dirs full, 0 dirs failed
W20250830 01:58:23.034917 27306 catalog_manager.cc:4908] TS 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): delete failed for tablet 177a8293c20c412ea767995c5c9af204 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 177a8293c20c412ea767995c5c9af204 already in progress: creating tablet
I20250830 01:58:23.035374 27632 tablet_service.cc:1468] Processing CreateTablet for tablet 47a4f27ca16d436386ed86810020f84b (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.035897 27632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a4f27ca16d436386ed86810020f84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:23.036074 27773 tablet_service.cc:1468] Processing CreateTablet for tablet 47a4f27ca16d436386ed86810020f84b (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.036646 27773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a4f27ca16d436386ed86810020f84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:23.038374 27318 catalog_manager.cc:2149] created HMS catalog entry for table [id=32148c9d0fcd4e8eb909814b954bc368]
I20250830 01:58:23.046371 27318 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "default.Table_2c93d305666e48d5b919f93ebe9e05ec" } new_table_name: "default.Table_942f583f6001419a91d24f8fdb74108c" modify_external_catalogs: true
W20250830 01:58:23.046463 27308 catalog_manager.cc:4908] TS 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297): delete failed for tablet 177a8293c20c412ea767995c5c9af204 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 177a8293c20c412ea767995c5c9af204 already in progress: creating tablet
W20250830 01:58:23.046778 27308 catalog_manager.cc:4908] TS 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297): delete failed for tablet fc86731b9fa247619aa9414e3709ba35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc86731b9fa247619aa9414e3709ba35 already in progress: creating tablet
I20250830 01:58:23.046983 27906 tablet_service.cc:1468] Processing CreateTablet for tablet 47a4f27ca16d436386ed86810020f84b (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250830 01:58:23.047096 27306 catalog_manager.cc:4908] TS 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): delete failed for tablet fc86731b9fa247619aa9414e3709ba35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc86731b9fa247619aa9414e3709ba35 already in progress: creating tablet
I20250830 01:58:23.047490 27906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a4f27ca16d436386ed86810020f84b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250830 01:58:23.048622 27307 catalog_manager.cc:4908] TS 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): delete failed for tablet 177a8293c20c412ea767995c5c9af204 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 177a8293c20c412ea767995c5c9af204 already in progress: creating tablet
W20250830 01:58:23.049669 27307 catalog_manager.cc:4908] TS 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): delete failed for tablet fc86731b9fa247619aa9414e3709ba35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc86731b9fa247619aa9414e3709ba35 already in progress: creating tablet
W20250830 01:58:23.053133 27316 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_badff657ae07402a88632f78ec1befb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:23.053442 27314 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_624ff0828e534d54a8ea5f69551f7e8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:23.054312 27313 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a36b1b6bfc854dfc907635b41d47be02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:23.054425 27315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_739ceee58b144d72b56342855499ecc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:23.055151 27312 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cef396d7e593458aa7679269acecf621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250830 01:58:23.055629 27320 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_11641b1a96984fa099ffa232b6a0033e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250830 01:58:23.058497 27631 tablet_service.cc:1468] Processing CreateTablet for tablet 779c87f5e2c34e8abd116c7c1c4f0092 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.059109 27631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779c87f5e2c34e8abd116c7c1c4f0092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:23.058995 27906 tablet_service.cc:1468] Processing CreateTablet for tablet 779c87f5e2c34e8abd116c7c1c4f0092 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.059758 27906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779c87f5e2c34e8abd116c7c1c4f0092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:23.066150 27975 heartbeater.cc:499] Master 127.26.105.254:36635 was elected leader, sending a full tablet report...
I20250830 01:58:23.069540 28048 ts_tablet_manager.cc:1428] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.083s user 0.015s sys 0.010s
I20250830 01:58:23.069610 27772 tablet_service.cc:1468] Processing CreateTablet for tablet 779c87f5e2c34e8abd116c7c1c4f0092 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:23.070641 27700 heartbeater.cc:499] Master 127.26.105.254:36635 was elected leader, sending a full tablet report...
I20250830 01:58:23.071789 28046 ts_tablet_manager.cc:1428] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.100s user 0.010s sys 0.003s
I20250830 01:58:23.072223 28048 tablet_bootstrap.cc:492] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:23.072867 27772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779c87f5e2c34e8abd116c7c1c4f0092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:23.074205 28048 tablet_bootstrap.cc:654] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.076020 27839 heartbeater.cc:499] Master 127.26.105.254:36635 was elected leader, sending a full tablet report...
I20250830 01:58:23.080760 28046 tablet_bootstrap.cc:492] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:23.081593 28058 raft_consensus.cc:491] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:23.081825 28058 raft_consensus.cc:513] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.085390 28047 ts_tablet_manager.cc:1428] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.131s user 0.019s sys 0.001s
I20250830 01:58:23.085922 28047 tablet_bootstrap.cc:492] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:23.087437 28046 tablet_bootstrap.cc:654] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.087917 28047 tablet_bootstrap.cc:654] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
F20250830 01:58:23.088351 27535 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 5 received events: NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_488d167c0efe4d1b91a27f563856f9f6, message={"server":"","servicePrincipal":"","db":"default","table":"table_488d167c0efe4d1b91a27f563856f9f6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_488d167c0efe4d1b91a27f563856f9f6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_488d167c0efe4d1b91a27f563856f9f6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_488d167c0efe4d1b91a27f563856f9f6\",\"kudu.table_id\":\"a66ba6403ba349f3ade7628ab65cd30f\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_badff657ae07402a88632f78ec1befb7, message={"server":"","servicePrincipal":"","db":"default","table":"table_badff657ae07402a88632f78ec1befb7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_badff657ae07402a88632f78ec1befb7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_badff657ae07402a88632f78ec1befb7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_badff657ae07402a88632f78ec1befb7\",\"kudu.table_id\":\"7b416700773a4e27bf152c666f48a272\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f978c1d4dcd google::LogMessage::Fail() at ??:0
@ 0x7f978c1d8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f978c1d47cc google::LogMessage::Flush() at ??:0
@ 0x7f978c1d4a11 google::LogMessage::~LogMessage() at ??:0
I20250830 01:58:23.095855 28048 tablet_bootstrap.cc:492] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:23.095781 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "177a8293c20c412ea767995c5c9af204" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:23.096060 28048 ts_tablet_manager.cc:1397] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.006s
I20250830 01:58:23.097200 28048 raft_consensus.cc:357] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
W20250830 01:58:23.097252 27727 leader_election.cc:343] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Illegal state: must be running to vote when last-logged opid is not known
I20250830 01:58:23.097460 28048 raft_consensus.cc:383] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.097537 28048 raft_consensus.cc:738] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.097846 28048 consensus_queue.cc:260] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.101126 28048 ts_tablet_manager.cc:1428] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250830 01:58:23.103888 28048 tablet_bootstrap.cc:492] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:23.105549 28048 tablet_bootstrap.cc:654] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.107398 28046 tablet_bootstrap.cc:492] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:23.107651 28046 ts_tablet_manager.cc:1397] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250830 01:58:23.108742 28046 raft_consensus.cc:357] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.108933 28046 raft_consensus.cc:383] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.109004 28046 raft_consensus.cc:738] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.109284 28046 consensus_queue.cc:260] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.109767 28046 ts_tablet_manager.cc:1428] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250830 01:58:23.111598 28046 tablet_bootstrap.cc:492] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:23.113487 28046 tablet_bootstrap.cc:654] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
W20250830 01:58:23.115541 27702 tablet.cc:2378] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250830 01:58:23.120465 28058 leader_election.cc:290] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.120502 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "177a8293c20c412ea767995c5c9af204" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:23.121093 27929 raft_consensus.cc:2466] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:23.121855 27729 leader_election.cc:304] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters: 03d61552c37a44ad9a7a6f6f5c236227
I20250830 01:58:23.122807 28058 raft_consensus.cc:2802] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:23.122977 28058 raft_consensus.cc:491] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:23.123085 28058 raft_consensus.cc:3058] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
@ 0x7f97a2609794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250830 01:58:23.132652 28048 tablet_bootstrap.cc:492] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:23.132848 28048 ts_tablet_manager.cc:1397] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.011s
I20250830 01:58:23.133716 28048 raft_consensus.cc:357] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.133916 28048 raft_consensus.cc:383] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.134008 28048 raft_consensus.cc:738] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.134402 28048 consensus_queue.cc:260] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.135795 28058 raft_consensus.cc:513] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.137157 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "177a8293c20c412ea767995c5c9af204" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:23.137459 27653 raft_consensus.cc:3058] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.138161 28048 ts_tablet_manager.cc:1428] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250830 01:58:23.141408 28048 tablet_bootstrap.cc:492] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:23.141878 27653 raft_consensus.cc:2466] T 177a8293c20c412ea767995c5c9af204 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.142540 27727 leader_election.cc:304] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:23.142933 28057 raft_consensus.cc:2802] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:23.143924 28046 tablet_bootstrap.cc:492] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:23.144110 28046 ts_tablet_manager.cc:1397] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.001s
I20250830 01:58:23.145133 28046 raft_consensus.cc:357] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.145368 28046 raft_consensus.cc:383] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.145468 28046 raft_consensus.cc:738] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.145820 28046 consensus_queue.cc:260] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.146600 28046 ts_tablet_manager.cc:1428] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250830 01:58:23.146986 28046 tablet_bootstrap.cc:492] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:23.148252 28058 leader_election.cc:290] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.148867 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "177a8293c20c412ea767995c5c9af204" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:23.149050 27929 raft_consensus.cc:3058] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.149281 28046 tablet_bootstrap.cc:654] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.150281 28048 tablet_bootstrap.cc:654] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.150559 28057 raft_consensus.cc:695] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:23.151124 28057 consensus_queue.cc:237] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.152729 27929 raft_consensus.cc:2466] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.154309 28047 tablet_bootstrap.cc:492] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:23.154479 28047 ts_tablet_manager.cc:1397] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.069s user 0.035s sys 0.016s
I20250830 01:58:23.155591 28047 raft_consensus.cc:357] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.155828 28047 raft_consensus.cc:383] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.155905 28047 raft_consensus.cc:738] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.156265 28047 consensus_queue.cc:260] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.156783 28047 ts_tablet_manager.cc:1428] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250830 01:58:23.157125 28047 tablet_bootstrap.cc:492] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:23.158717 28056 raft_consensus.cc:491] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:23.159022 28047 tablet_bootstrap.cc:654] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.158890 28056 raft_consensus.cc:513] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.160962 28056 leader_election.cc:290] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.163699 28047 tablet_bootstrap.cc:492] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:23.163859 28047 ts_tablet_manager.cc:1397] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250830 01:58:23.164876 28047 raft_consensus.cc:357] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.165077 28047 raft_consensus.cc:383] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.165174 28047 raft_consensus.cc:738] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.165439 28047 consensus_queue.cc:260] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.167534 28047 ts_tablet_manager.cc:1428] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250830 01:58:23.168818 28047 tablet_bootstrap.cc:492] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:23.170720 28047 tablet_bootstrap.cc:654] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.171409 27793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3996bdda030340d081af363dbdfea328" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5563a27e4e4748813d624aab13cdca" is_pre_election: true
W20250830 01:58:23.172364 27588 leader_election.cc:343] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Illegal state: must be running to vote when last-logged opid is not known
I20250830 01:58:23.174813 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3996bdda030340d081af363dbdfea328" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
@ 0x7f97a2608041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
W20250830 01:58:23.175452 27589 leader_election.cc:343] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297): Illegal state: must be running to vote when last-logged opid is not known
I20250830 01:58:23.175585 27589 leader_election.cc:304] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227; no voters: 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822
I20250830 01:58:23.176419 28056 raft_consensus.cc:2747] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250830 01:58:23.175756 28048 tablet_bootstrap.cc:492] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:23.178614 28048 ts_tablet_manager.cc:1397] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.004s
I20250830 01:58:23.179759 28048 raft_consensus.cc:357] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.180449 28048 raft_consensus.cc:383] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.180550 28048 raft_consensus.cc:738] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.180866 28048 consensus_queue.cc:260] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
@ 0x7f97a260fcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250830 01:58:23.181571 28048 ts_tablet_manager.cc:1428] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250830 01:58:23.181926 28048 tablet_bootstrap.cc:492] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:23.184062 28048 tablet_bootstrap.cc:654] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.187359 28046 tablet_bootstrap.cc:492] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:23.187542 28046 ts_tablet_manager.cc:1397] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20250830 01:58:23.188392 28046 raft_consensus.cc:357] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.188589 28046 raft_consensus.cc:383] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.188664 28046 raft_consensus.cc:738] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.188935 28046 consensus_queue.cc:260] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.191455 28046 ts_tablet_manager.cc:1428] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250830 01:58:23.192746 28046 tablet_bootstrap.cc:492] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:23.194984 28046 tablet_bootstrap.cc:654] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.197592 28047 tablet_bootstrap.cc:492] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:23.197748 28047 ts_tablet_manager.cc:1397] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.016s
I20250830 01:58:23.198570 28047 raft_consensus.cc:357] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.198762 28047 raft_consensus.cc:383] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.198825 28047 raft_consensus.cc:738] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.199050 28047 consensus_queue.cc:260] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
W20250830 01:58:23.199330 27977 tablet.cc:2378] T 177a8293c20c412ea767995c5c9af204 P 893b0b641a51432f9172ed0b41521822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250830 01:58:23.199594 28047 ts_tablet_manager.cc:1428] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250830 01:58:23.200098 28047 tablet_bootstrap.cc:492] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:23.201872 28047 tablet_bootstrap.cc:654] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
W20250830 01:58:23.208640 27846 tablet.cc:2378] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250830 01:58:23.211282 28048 tablet_bootstrap.cc:492] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:23.211444 28048 ts_tablet_manager.cc:1397] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.010s
I20250830 01:58:23.212301 28048 raft_consensus.cc:357] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.212477 28048 raft_consensus.cc:383] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.212558 28048 raft_consensus.cc:738] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.213311 28046 tablet_bootstrap.cc:492] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:23.213455 28046 ts_tablet_manager.cc:1397] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250830 01:58:23.214218 28046 raft_consensus.cc:357] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.214396 28046 raft_consensus.cc:383] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.214483 28046 raft_consensus.cc:738] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.214721 28046 consensus_queue.cc:260] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.215030 28048 consensus_queue.cc:260] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.217227 28046 ts_tablet_manager.cc:1428] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250830 01:58:23.219532 28048 ts_tablet_manager.cc:1428] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250830 01:58:23.223160 28046 tablet_bootstrap.cc:492] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:23.224964 28046 tablet_bootstrap.cc:654] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.226121 28047 tablet_bootstrap.cc:492] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:23.226320 28047 ts_tablet_manager.cc:1397] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.008s
I20250830 01:58:23.227140 28047 raft_consensus.cc:357] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.227310 28047 raft_consensus.cc:383] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.227386 28047 raft_consensus.cc:738] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.227627 28047 consensus_queue.cc:260] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.227818 28048 tablet_bootstrap.cc:492] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:23.229507 28048 tablet_bootstrap.cc:654] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.229981 28047 ts_tablet_manager.cc:1428] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250830 01:58:23.232434 28047 tablet_bootstrap.cc:492] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:23.234109 28047 tablet_bootstrap.cc:654] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:23.236269 28046 tablet_bootstrap.cc:492] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:23.236449 28046 ts_tablet_manager.cc:1397] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20250830 01:58:23.237248 28046 raft_consensus.cc:357] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.237452 28046 raft_consensus.cc:383] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.237524 28046 raft_consensus.cc:738] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.237802 28046 consensus_queue.cc:260] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
@ 0x7f97a260fb39 std::_Function_handler<>::_M_invoke() at ??:0
I20250830 01:58:23.241487 28046 ts_tablet_manager.cc:1428] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250830 01:58:23.242648 28048 tablet_bootstrap.cc:492] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:23.242813 28048 ts_tablet_manager.cc:1397] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250830 01:58:23.243675 28048 raft_consensus.cc:357] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.243913 28048 raft_consensus.cc:383] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.243997 28048 raft_consensus.cc:738] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.244272 28048 consensus_queue.cc:260] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.245249 28048 ts_tablet_manager.cc:1428] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
@ 0xa774dd std::function<>::operator()() at ??:?
I20250830 01:58:23.263506 28047 tablet_bootstrap.cc:492] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:23.263700 28047 ts_tablet_manager.cc:1397] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.019s
I20250830 01:58:23.264571 28047 raft_consensus.cc:357] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.264839 28047 raft_consensus.cc:383] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:23.264917 28047 raft_consensus.cc:738] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.265239 28047 consensus_queue.cc:260] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.269869 28047 ts_tablet_manager.cc:1428] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
@ 0x7f978ef31cc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f97923536db start_thread at ??:0
@ 0x7f978b08571f clone at ??:0
I20250830 01:58:23.330404 28057 raft_consensus.cc:491] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:23.330628 28057 raft_consensus.cc:513] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.331365 28057 leader_election.cc:290] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.332173 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3996bdda030340d081af363dbdfea328" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:23.332377 27653 raft_consensus.cc:2466] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:23.333110 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3996bdda030340d081af363dbdfea328" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:23.333300 27929 raft_consensus.cc:2466] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:23.333858 27727 leader_election.cc:304] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:23.334376 28057 raft_consensus.cc:2802] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:23.334460 28057 raft_consensus.cc:491] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:23.334540 28057 raft_consensus.cc:3058] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.337178 28057 raft_consensus.cc:513] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.338330 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3996bdda030340d081af363dbdfea328" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:23.338533 27653 raft_consensus.cc:3058] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.341565 27653 raft_consensus.cc:2466] T 3996bdda030340d081af363dbdfea328 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.342033 27727 leader_election.cc:304] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:23.342399 28066 raft_consensus.cc:2802] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:23.342531 28066 raft_consensus.cc:695] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:23.342835 28066 consensus_queue.cc:237] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.346529 28057 leader_election.cc:290] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.347541 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--webserver_interface=127.26.105.252
--webserver_port=39345
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250830 01:58:23.348308 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3996bdda030340d081af363dbdfea328" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:23.348517 27929 raft_consensus.cc:3058] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.351761 27929 raft_consensus.cc:2466] T 3996bdda030340d081af363dbdfea328 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.370736 28057 raft_consensus.cc:491] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:23.370971 28057 raft_consensus.cc:513] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.371817 28057 leader_election.cc:290] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.373025 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "779c87f5e2c34e8abd116c7c1c4f0092" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:23.373304 27653 raft_consensus.cc:2466] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:23.373696 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "779c87f5e2c34e8abd116c7c1c4f0092" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:23.373929 27929 raft_consensus.cc:2466] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:23.374447 27729 leader_election.cc:304] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters:
I20250830 01:58:23.374912 28057 raft_consensus.cc:2802] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:23.374996 28057 raft_consensus.cc:491] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:23.375077 28057 raft_consensus.cc:3058] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.377777 28057 raft_consensus.cc:513] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.379195 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "779c87f5e2c34e8abd116c7c1c4f0092" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:23.379415 27653 raft_consensus.cc:3058] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.382086 27653 raft_consensus.cc:2466] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.384171 27727 leader_election.cc:304] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:23.384496 28066 raft_consensus.cc:2802] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:23.384629 28066 raft_consensus.cc:695] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:23.384876 28066 consensus_queue.cc:237] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.388450 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "779c87f5e2c34e8abd116c7c1c4f0092" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:23.388685 27929 raft_consensus.cc:3058] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.391497 27929 raft_consensus.cc:2466] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.392375 28057 leader_election.cc:290] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.416270 28056 raft_consensus.cc:491] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:23.416509 28056 raft_consensus.cc:513] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.417326 28056 leader_election.cc:290] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.418447 27793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc86731b9fa247619aa9414e3709ba35" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5563a27e4e4748813d624aab13cdca" is_pre_election: true
I20250830 01:58:23.418766 27793 raft_consensus.cc:2466] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 0.
I20250830 01:58:23.419363 27588 leader_election.cc:304] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:23.420214 28056 raft_consensus.cc:2802] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:23.420387 28056 raft_consensus.cc:491] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:23.420491 28056 raft_consensus.cc:3058] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.423604 28056 raft_consensus.cc:513] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.424870 27793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc86731b9fa247619aa9414e3709ba35" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5563a27e4e4748813d624aab13cdca"
I20250830 01:58:23.425078 27793 raft_consensus.cc:3058] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.428014 27793 raft_consensus.cc:2466] T fc86731b9fa247619aa9414e3709ba35 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 1.
I20250830 01:58:23.428414 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc86731b9fa247619aa9414e3709ba35" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:23.428653 27929 raft_consensus.cc:2466] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 0.
I20250830 01:58:23.429737 27588 leader_election.cc:304] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:23.431875 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc86731b9fa247619aa9414e3709ba35" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:23.432121 27929 raft_consensus.cc:3058] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.434449 28056 leader_election.cc:290] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 election: Requested vote from peers 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.434896 28056 raft_consensus.cc:2802] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:23.435303 28056 raft_consensus.cc:695] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 LEADER]: Becoming Leader. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Running, Role: LEADER
I20250830 01:58:23.436040 28056 consensus_queue.cc:237] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:23.437371 27929 raft_consensus.cc:2466] T fc86731b9fa247619aa9414e3709ba35 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 1.
01:58:23.463 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_badff657ae07402a88632f78ec1befb7 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I20250830 01:58:23.557555 28095 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:23.557776 28095 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:23.558539 28095 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:23.561081 27377 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:23.608641 27985 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.105.254:36635)
W20250830 01:58:23.608888 27378 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:23.609041 27985 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.26.105.254:36635)
W20250830 01:58:23.609895 27985 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.26.105.254:36635)
W20250830 01:58:23.614933 27378 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:23.615196 27378 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:23.615906 28095 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250830 01:58:23.721807 28055 raft_consensus.cc:491] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:23.722034 28055 raft_consensus.cc:513] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.723819 28055 leader_election.cc:290] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:23.745354 27793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddb4c87468a44d1a2bae0fbd6784a36" candidate_uuid: "893b0b641a51432f9172ed0b41521822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5563a27e4e4748813d624aab13cdca" is_pre_election: true
I20250830 01:58:23.745357 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddb4c87468a44d1a2bae0fbd6784a36" candidate_uuid: "893b0b641a51432f9172ed0b41521822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:23.745585 27793 raft_consensus.cc:2466] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 893b0b641a51432f9172ed0b41521822 in term 0.
I20250830 01:58:23.745622 27653 raft_consensus.cc:2466] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 893b0b641a51432f9172ed0b41521822 in term 0.
I20250830 01:58:23.746206 27864 leader_election.cc:304] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters:
I20250830 01:58:23.746589 28055 raft_consensus.cc:2802] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:23.746724 28055 raft_consensus.cc:491] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:23.746837 28055 raft_consensus.cc:3058] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.748992 28055 raft_consensus.cc:513] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.749684 28055 leader_election.cc:290] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:23.750110 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddb4c87468a44d1a2bae0fbd6784a36" candidate_uuid: "893b0b641a51432f9172ed0b41521822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:23.750335 27653 raft_consensus.cc:3058] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.750376 27793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddb4c87468a44d1a2bae0fbd6784a36" candidate_uuid: "893b0b641a51432f9172ed0b41521822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5563a27e4e4748813d624aab13cdca"
I20250830 01:58:23.750573 27793 raft_consensus.cc:3058] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.752350 27793 raft_consensus.cc:2466] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 893b0b641a51432f9172ed0b41521822 in term 1.
I20250830 01:58:23.752704 27864 leader_election.cc:304] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters:
I20250830 01:58:23.752962 28055 raft_consensus.cc:2802] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:23.752974 27653 raft_consensus.cc:2466] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 893b0b641a51432f9172ed0b41521822 in term 1.
W20250830 01:58:23.753551 28091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:23.753836 28091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:23.753960 28091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250830 01:58:23.754398 28055 raft_consensus.cc:695] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [term 1 LEADER]: Becoming Leader. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Running, Role: LEADER
I20250830 01:58:23.754961 28055 consensus_queue.cc:237] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
W20250830 01:58:23.764530 28091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:23.764627 28091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:23.764695 28091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:23.764778 28091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:23.774863 28091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.105.252
--webserver_port=39345
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:23.776753 28091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:23.778669 28091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:23.786434 28110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:23.786332 28109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:23.786756 28066 raft_consensus.cc:491] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250830 01:58:23.787034 28112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:23.786954 28066 raft_consensus.cc:513] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.787734 28066 leader_election.cc:290] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.787853 28091 server_base.cc:1047] running on GCE node
I20250830 01:58:23.788268 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a4f27ca16d436386ed86810020f84b" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:23.788543 27929 raft_consensus.cc:2466] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:23.788899 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a4f27ca16d436386ed86810020f84b" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:23.788945 27729 leader_election.cc:304] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters:
I20250830 01:58:23.789109 27653 raft_consensus.cc:2466] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:23.789227 28066 raft_consensus.cc:2802] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:23.789227 28091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:23.789327 28066 raft_consensus.cc:491] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:23.789415 28066 raft_consensus.cc:3058] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.790778 28091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:23.791733 28066 raft_consensus.cc:513] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.791973 28091 hybrid_clock.cc:648] HybridClock initialized: now 1756519103791921 us; error 44 us; skew 500 ppm
I20250830 01:58:23.792265 28066 leader_election.cc:290] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:23.792308 28091 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:23.792682 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a4f27ca16d436386ed86810020f84b" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:23.792852 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a4f27ca16d436386ed86810020f84b" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:23.792923 27653 raft_consensus.cc:3058] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.793062 27929 raft_consensus.cc:3058] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:23.795883 27929 raft_consensus.cc:2466] T 47a4f27ca16d436386ed86810020f84b P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.795883 27653 raft_consensus.cc:2466] T 47a4f27ca16d436386ed86810020f84b P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:23.796398 27727 leader_election.cc:304] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:23.796691 28066 raft_consensus.cc:2802] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:23.796825 28066 raft_consensus.cc:695] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:23.797137 28066 consensus_queue.cc:237] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:23.797885 28091 webserver.cc:480] Webserver started at http://127.26.105.252:39345/ using document root <none> and password file <none>
I20250830 01:58:23.798913 28091 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:23.799026 28091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:23.805141 28091 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250830 01:58:23.808393 28119 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:23.809787 28091 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250830 01:58:23.809949 28091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
uuid: "fd4f5c1e7015460e99715d8a2edd2959"
format_stamp: "Formatted at 2025-08-30 01:58:20 on dist-test-slave-gp2r"
I20250830 01:58:23.810663 28091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:23.832993 28091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:23.833889 28091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:23.834220 28091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:23.854982 28091 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.252:46233
I20250830 01:58:23.855005 28170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.252:46233 every 8 connection(s)
I20250830 01:58:23.856631 28091 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
I20250830 01:58:23.860373 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 28091
I20250830 01:58:23.866588 28171 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:23.872954 28171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap starting.
I20250830 01:58:23.883090 28171 log.cc:826] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:23.891216 28171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250830 01:58:23.891716 28171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap complete.
I20250830 01:58:23.896818 28171 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:23.897491 28171 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4f5c1e7015460e99715d8a2edd2959, State: Initialized, Role: FOLLOWER
I20250830 01:58:23.898087 28171 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:23.898943 28176 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:23.899190 28176 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:23.900216 28171 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:23.912261 28190 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:23.912370 28190 catalog_manager.cc:1554] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: loading cluster ID for follower catalog manager: success
I20250830 01:58:23.909744 28091 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:58:23.916311 28091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:23.916621 28190 catalog_manager.cc:1576] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: acquiring CA information for follower catalog manager: success
I20250830 01:58:23.919370 28190 catalog_manager.cc:1604] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:23.959908 27699 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:23.963057 28135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:40217; Asking this server to re-register.
I20250830 01:58:23.963822 27699 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:23.965368 28135 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:24.142936 28066 consensus_queue.cc:1035] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.153733 27974 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:24.153957 27974 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
I20250830 01:58:24.159469 28135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:48511; Asking this server to re-register.
I20250830 01:58:24.161223 28057 consensus_queue.cc:1035] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.166914 27974 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:24.169324 28135 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:24.177140 27838 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:24.179453 28135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:46181; Asking this server to re-register.
I20250830 01:58:24.180773 27838 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:24.182607 28135 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:24.199622 28066 consensus_queue.cc:1035] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.208891 28057 consensus_queue.cc:1035] T 3996bdda030340d081af363dbdfea328 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.250757 28066 consensus_queue.cc:1035] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.253612 28198 consensus_queue.cc:1035] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.261174 28066 consensus_queue.cc:1035] T 177a8293c20c412ea767995c5c9af204 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.282034 28198 consensus_queue.cc:1035] T fc86731b9fa247619aa9414e3709ba35 P 03d61552c37a44ad9a7a6f6f5c236227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.290445 28212 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:24.290711 28212 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:24.291785 28212 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:24.303750 27378 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:24.305099 28146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "fd4f5c1e7015460e99715d8a2edd2959" is_pre_election: true
I20250830 01:58:24.305660 28146 raft_consensus.cc:2466] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba697035f304124a88a1cf1aabb748e in term 1.
I20250830 01:58:24.306124 27377 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959; no voters: ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:24.306475 28212 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250830 01:58:24.306597 28212 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:24.306690 28212 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 2 FOLLOWER]: Advancing to term 3
I20250830 01:58:24.309772 28212 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:24.310487 28212 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 election: Requested vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:24.311159 28146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "fd4f5c1e7015460e99715d8a2edd2959"
I20250830 01:58:24.311396 28146 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 1 FOLLOWER]: Advancing to term 3
W20250830 01:58:24.312158 27378 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:24.315378 28146 raft_consensus.cc:2466] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba697035f304124a88a1cf1aabb748e in term 3.
I20250830 01:58:24.315861 27377 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959; no voters: ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:24.316236 28212 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 FOLLOWER]: Leader election won for term 3
I20250830 01:58:24.316733 28212 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 LEADER]: Becoming Leader. State: Replica: 9ba697035f304124a88a1cf1aabb748e, State: Running, Role: LEADER
I20250830 01:58:24.317183 28212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:24.318845 28216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ba697035f304124a88a1cf1aabb748e. Latest consensus state: current_term: 3 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:24.319077 28216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:24.321470 28218 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:24.323982 28218 catalog_manager.cc:671] Loaded metadata for table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]
I20250830 01:58:24.324414 28218 catalog_manager.cc:671] Loaded metadata for table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]
I20250830 01:58:24.324762 28218 catalog_manager.cc:671] Loaded metadata for table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]
I20250830 01:58:24.325098 28218 catalog_manager.cc:671] Loaded metadata for table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]
I20250830 01:58:24.326912 28218 tablet_loader.cc:96] loaded metadata for tablet 3996bdda030340d081af363dbdfea328 (table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46])
I20250830 01:58:24.327248 28218 tablet_loader.cc:96] loaded metadata for tablet 47a4f27ca16d436386ed86810020f84b (table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f])
I20250830 01:58:24.327543 28218 tablet_loader.cc:96] loaded metadata for tablet 779c87f5e2c34e8abd116c7c1c4f0092 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368])
I20250830 01:58:24.327852 28218 tablet_loader.cc:96] loaded metadata for tablet cddb4c87468a44d1a2bae0fbd6784a36 (table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a])
I20250830 01:58:24.328275 28218 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:24.329149 28218 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:24.329231 28218 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:24.330411 28218 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:24.331236 28218 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Loaded TSK: 0
I20250830 01:58:24.332044 28218 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:24.332845 28218 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:24.333590 28218 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250830 01:58:24.345016 27391 master_service.cc:948] ReplaceTablet: received request to replace tablet 3996bdda030340d081af363dbdfea328 from {username='slave'} at 127.0.0.1:46212
I20250830 01:58:24.345201 27389 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:46212
I20250830 01:58:24.345758 27387 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:46212
I20250830 01:58:24.347173 27383 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:46212
I20250830 01:58:24.349113 27389 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:46212
I20250830 01:58:24.345050 27390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "default.Table_1ad78e45bc5842aeaf69a74b3ef38007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:24.347738 27387 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "default.Table_624ff0828e534d54a8ea5f69551f7e8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:24.346457 27385 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46212:
table { table_name: "default.Table_d19bb3dab12e478cbd8897446f84d508" } new_table_name: "default.Table_31e5e45019fd41888d6a471fbd3736b4" modify_external_catalogs: true
I20250830 01:58:24.346928 27384 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "default.Table_739ceee58b144d72b56342855499ecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:24.345791 27386 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "default.Table_11641b1a96984fa099ffa232b6a0033e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:24.348038 27383 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "default.Table_a36b1b6bfc854dfc907635b41d47be02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:24.348279 27382 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "default.Table_cef396d7e593458aa7679269acecf621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:24.349766 27389 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46212:
table { table_name: "default.Table_488d167c0efe4d1b91a27f563856f9f6" } new_table_name: "default.Table_59d8d3858b8c490f984614c9187eef7c" modify_external_catalogs: true
I20250830 01:58:24.345314 27388 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "default.Table_badff657ae07402a88632f78ec1befb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:24.357035 28146 raft_consensus.cc:1273] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Refusing update from remote peer 9ba697035f304124a88a1cf1aabb748e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20250830 01:58:24.357937 28216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 4, Time since last communication: 0.000s
W20250830 01:58:24.359177 27378 consensus_peers.cc:489] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e -> Peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Couldn't send request to peer ad9824f7e6604077ac8a034879b4c731. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250830 01:58:24.362565 27553 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 5 received events: NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_488d167c0efe4d1b91a27f563856f9f6, message={"server":"","servicePrincipal":"","db":"default","table":"table_488d167c0efe4d1b91a27f563856f9f6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_488d167c0efe4d1b91a27f563856f9f6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_488d167c0efe4d1b91a27f563856f9f6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_488d167c0efe4d1b91a27f563856f9f6\",\"kudu.table_id\":\"a66ba6403ba349f3ade7628ab65cd30f\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_badff657ae07402a88632f78ec1befb7, message={"server":"","servicePrincipal":"","db":"default","table":"table_badff657ae07402a88632f78ec1befb7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_badff657ae07402a88632f78ec1befb7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_badff657ae07402a88632f78ec1befb7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_badff657ae07402a88632f78ec1befb7\",\"kudu.table_id\":\"7b416700773a4e27bf152c666f48a272\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f7d49fc2dcd google::LogMessage::Fail() at ??:0
@ 0x7f7d49fc6b93 google::LogMessage::SendToLog() at ??:0
I20250830 01:58:24.367622 28176 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ba697035f304124a88a1cf1aabb748e. Latest consensus state: current_term: 3 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:24.367815 28176 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f7d49fc27cc google::LogMessage::Flush() at ??:0
I20250830 01:58:24.368211 28223 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:24.368621 28223 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f7d49fc2a11 google::LogMessage::~LogMessage() at ??:0
I20250830 01:58:24.371502 28212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:24.371734 28212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:24.372689 28223 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:24.372900 28223 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:24.377525 28220 mvcc.cc:204] Tried to move back new op lower bound from 7194702246100099072 to 7194702244013260800. Current Snapshot: MvccSnapshot[applied={T|T < 7194702246100099072}]
@ 0x7f7d603f7794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250830 01:58:24.379215 27391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Sending DeleteTablet for 3 replicas of tablet 3996bdda030340d081af363dbdfea328
I20250830 01:58:24.381040 27391 catalog_manager.cc:6652] ReplaceTablet: tablet 3996bdda030340d081af363dbdfea328 deleted and replaced by tablet 876295e43d7c4b01b2f772a492bc27cf
@ 0x7f7d603f6041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250830 01:58:24.390278 27391 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46212:
table { table_name: "default.Table_2c4fc026b80045d8a58497a870104278" } new_table_name: "default.Table_f8d128ad097b4e25a4cc03e5b2f02676" modify_external_catalogs: true
@ 0x7f7d603fdcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7d603fdb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f7d4cd1fcc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7d501416db start_thread at ??:0
@ 0x7f7d48e7371f clone at ??:0
W20250830 01:58:24.580518 27984 connection.cc:537] client connection to 127.26.105.253:39521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:24.581591 28231 negotiation.cc:337] Failed RPC negotiation. Trace:
0830 01:58:24.382118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.26.105.194:43787)
0830 01:58:24.407767 (+ 25649us) server_negotiation.cc:206] Beginning negotiation
0830 01:58:24.407775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0830 01:58:24.407802 (+ 27us) server_negotiation.cc:407] Connection header received
0830 01:58:24.407883 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0830 01:58:24.407892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0830 01:58:24.407973 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0830 01:58:24.408075 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0830 01:58:24.580614 (+172539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54164: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":25505,"thread_start_us":25379,"threads_started":1}
W20250830 01:58:24.581614 28232 negotiation.cc:337] Failed RPC negotiation. Trace:
0830 01:58:24.383059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.26.105.195:35297)
0830 01:58:24.408572 (+ 25513us) server_negotiation.cc:206] Beginning negotiation
0830 01:58:24.408578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0830 01:58:24.580567 (+171989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43552: BlockingRecv error: recv got EOF from 127.0.0.1:43552 (error 108)
Metrics: {"server-negotiator.queue_time_us":25398,"thread_start_us":25280,"threads_started":1}
W20250830 01:58:24.582276 28229 negotiation.cc:337] Failed RPC negotiation. Trace:
0830 01:58:24.381311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.26.105.193:36219)
0830 01:58:24.408960 (+ 27649us) server_negotiation.cc:206] Beginning negotiation
0830 01:58:24.408965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0830 01:58:24.408986 (+ 21us) server_negotiation.cc:407] Connection header received
0830 01:58:24.409065 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0830 01:58:24.409072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0830 01:58:24.409142 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0830 01:58:24.409220 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0830 01:58:24.581444 (+172224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45224: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":27563,"thread_start_us":27458,"threads_started":1}
I20250830 01:58:24.635138 28206 consensus_queue.cc:1035] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.639807 28206 consensus_queue.cc:1035] T 47a4f27ca16d436386ed86810020f84b P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.737946 28055 consensus_queue.cc:1035] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.744594 28250 consensus_queue.cc:1035] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:24.905829 28253 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:24.906090 28253 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:24.908151 28253 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:24.908936 28123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:24.910248 28122 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:24.910512 28123 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:24.910712 28123 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:24.911170 28253 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250830 01:58:25.642513 28262 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:25.642719 28262 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:25.643393 28262 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:25.646420 28123 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:25.646591 28122 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:25.646710 28122 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:25.647089 28262 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250830 01:58:26.726739 28286 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:26.726933 28286 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:26.727692 28286 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:26.729771 28122 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:26.729976 28123 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:26.730111 28123 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:26.730437 28286 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250830 01:58:27.149152 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--webserver_interface=127.26.105.254
--webserver_port=37963
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:27.490810 28295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:27.491163 28295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:27.491286 28295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:27.499656 28295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:27.499770 28295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:27.499855 28295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:27.499922 28295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:27.510310 28295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.105.254
--webserver_port=37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:27.511967 28295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:27.514210 28295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:27.522626 28307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:27.522670 28308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:27.522584 28310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:27.523494 28295 server_base.cc:1047] running on GCE node
I20250830 01:58:27.524221 28295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:27.525619 28295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:27.526818 28295 hybrid_clock.cc:648] HybridClock initialized: now 1756519107526772 us; error 36 us; skew 500 ppm
I20250830 01:58:27.527171 28295 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:27.529354 28295 webserver.cc:480] Webserver started at http://127.26.105.254:37963/ using document root <none> and password file <none>
I20250830 01:58:27.530107 28295 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:27.530246 28295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:27.535486 28295 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250830 01:58:27.538461 28316 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:27.539861 28295 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250830 01:58:27.540038 28295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
uuid: "ad9824f7e6604077ac8a034879b4c731"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:27.540731 28295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:27.562994 28295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:27.563898 28295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:27.564184 28295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:27.584162 28295 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.254:36635
I20250830 01:58:27.584192 28367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.254:36635 every 8 connection(s)
I20250830 01:58:27.585788 28295 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
I20250830 01:58:27.589051 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 28295
I20250830 01:58:27.593662 28368 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:27.598731 28368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap starting.
I20250830 01:58:27.610006 28368 log.cc:826] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:27.626365 28368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=13 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250830 01:58:27.626886 28368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap complete.
I20250830 01:58:27.631947 28368 raft_consensus.cc:357] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:27.632778 28368 raft_consensus.cc:738] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Initialized, Role: FOLLOWER
I20250830 01:58:27.633419 28368 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:27.634258 28373 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:27.634543 28373 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:27.635255 28368 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:27.658244 28387 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:27.658365 28387 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: loading cluster ID for follower catalog manager: success
I20250830 01:58:27.665689 28387 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: acquiring CA information for follower catalog manager: success
I20250830 01:58:27.669677 28387 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:27.690306 28295 master_runner.cc:186] Error getting master registration for 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111),
I20250830 01:58:27.696803 28295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:27.697064 27700 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:27.697281 27700 heartbeater.cc:499] Master 127.26.105.254:36635 was elected leader, sending a full tablet report...
I20250830 01:58:27.700891 28333 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:41517; Asking this server to re-register.
I20250830 01:58:27.701902 27700 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:27.703936 28333 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:27.874735 27839 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:27.874960 27839 heartbeater.cc:499] Master 127.26.105.254:36635 was elected leader, sending a full tablet report...
I20250830 01:58:27.876890 28333 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:46807; Asking this server to re-register.
I20250830 01:58:27.877545 27839 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:27.878881 28333 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:27.901026 27975 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:27.901216 27975 heartbeater.cc:499] Master 127.26.105.254:36635 was elected leader, sending a full tablet report...
I20250830 01:58:27.903205 28333 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:58547; Asking this server to re-register.
I20250830 01:58:27.903898 27975 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:27.905303 28333 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:28.143483 28398 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:28.143779 28398 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:28.145465 28398 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:28.146243 28319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:28.147718 28319 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:28.151938 28146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "fd4f5c1e7015460e99715d8a2edd2959" is_pre_election: true
I20250830 01:58:28.152212 28146 raft_consensus.cc:2391] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad9824f7e6604077ac8a034879b4c731 in current term 3: Already voted for candidate 9ba697035f304124a88a1cf1aabb748e in this term.
I20250830 01:58:28.152793 28319 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:28.153199 28398 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 2 FOLLOWER]: Advancing to term 3
I20250830 01:58:28.157078 28398 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250830 01:58:28.326370 28401 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:28.326612 28401 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:28.327651 28401 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:28.330366 28122 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:28.335342 28343 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:28.335855 28343 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd4f5c1e7015460e99715d8a2edd2959 in term 3.
I20250830 01:58:28.336583 28123 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e
I20250830 01:58:28.336941 28401 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250830 01:58:28.337052 28401 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:28.337147 28401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 3 FOLLOWER]: Advancing to term 4
I20250830 01:58:28.339221 28401 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:28.339831 28401 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 election: Requested vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
I20250830 01:58:28.340332 28343 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731"
I20250830 01:58:28.340515 28343 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 3 FOLLOWER]: Advancing to term 4
W20250830 01:58:28.341053 28122 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:28.342608 28343 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd4f5c1e7015460e99715d8a2edd2959 in term 4.
I20250830 01:58:28.343005 28123 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e
I20250830 01:58:28.343328 28401 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 FOLLOWER]: Leader election won for term 4
I20250830 01:58:28.343768 28401 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 LEADER]: Becoming Leader. State: Replica: fd4f5c1e7015460e99715d8a2edd2959, State: Running, Role: LEADER
I20250830 01:58:28.344205 28401 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:28.345852 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd4f5c1e7015460e99715d8a2edd2959. Latest consensus state: current_term: 4 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:28.346158 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:28.346964 28407 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:28.349576 28407 catalog_manager.cc:671] Loaded metadata for table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]
I20250830 01:58:28.350499 28407 catalog_manager.cc:671] Loaded metadata for table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]
I20250830 01:58:28.350952 28407 catalog_manager.cc:671] Loaded metadata for table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]
I20250830 01:58:28.351301 28407 catalog_manager.cc:671] Loaded metadata for table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]
I20250830 01:58:28.353610 28407 tablet_loader.cc:96] loaded metadata for tablet 47a4f27ca16d436386ed86810020f84b (table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f])
I20250830 01:58:28.353930 28407 tablet_loader.cc:96] loaded metadata for tablet 779c87f5e2c34e8abd116c7c1c4f0092 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368])
I20250830 01:58:28.354215 28407 tablet_loader.cc:96] loaded metadata for tablet 876295e43d7c4b01b2f772a492bc27cf (table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46])
I20250830 01:58:28.354532 28407 tablet_loader.cc:96] loaded metadata for tablet cddb4c87468a44d1a2bae0fbd6784a36 (table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a])
I20250830 01:58:28.354930 28407 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:28.355821 28407 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:28.355895 28407 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:28.357048 28407 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:28.357847 28407 catalog_manager.cc:5966] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Loaded TSK: 0
I20250830 01:58:28.358760 28407 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:28.359557 28407 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:28.360294 28407 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250830 01:58:28.367338 28135 master_service.cc:948] ReplaceTablet: received request to replace tablet 3996bdda030340d081af363dbdfea328 from {username='slave'} at 127.0.0.1:39492
I20250830 01:58:28.367645 28134 master_service.cc:948] ReplaceTablet: received request to replace tablet cddb4c87468a44d1a2bae0fbd6784a36 from {username='slave'} at 127.0.0.1:39492
I20250830 01:58:28.368083 28136 master_service.cc:948] ReplaceTablet: received request to replace tablet cddb4c87468a44d1a2bae0fbd6784a36 from {username='slave'} at 127.0.0.1:39492
I20250830 01:58:28.368275 28133 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:39492
I20250830 01:58:28.369345 28132 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_ac070120b97141e188efab1e24ece67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.370958 28135 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_d4634c7907fc46558619aee806e21da3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.370878 28130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_badff657ae07402a88632f78ec1befb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.371083 28131 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_509af6139e2e4ba6bc5c537af315df6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.371574 28129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_1ad78e45bc5842aeaf69a74b3ef38007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.372295 28128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39492:
table { table_name: "default.Table_d19bb3dab12e478cbd8897446f84d508" } new_table_name: "default.Table_31e5e45019fd41888d6a471fbd3736b4" modify_external_catalogs: true
I20250830 01:58:28.372756 28127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_739ceee58b144d72b56342855499ecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.373485 28133 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_11641b1a96984fa099ffa232b6a0033e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.378779 28343 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Refusing update from remote peer fd4f5c1e7015460e99715d8a2edd2959: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 4 index: 22. (index mismatch)
F20250830 01:58:28.378906 28189 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 5 received events: NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_488d167c0efe4d1b91a27f563856f9f6, message={"server":"","servicePrincipal":"","db":"default","table":"table_488d167c0efe4d1b91a27f563856f9f6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_488d167c0efe4d1b91a27f563856f9f6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_488d167c0efe4d1b91a27f563856f9f6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_488d167c0efe4d1b91a27f563856f9f6\",\"kudu.table_id\":\"a66ba6403ba349f3ade7628ab65cd30f\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_badff657ae07402a88632f78ec1befb7, message={"server":"","servicePrincipal":"","db":"default","table":"table_badff657ae07402a88632f78ec1befb7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_badff657ae07402a88632f78ec1befb7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_badff657ae07402a88632f78ec1befb7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_badff657ae07402a88632f78ec1befb7\",\"kudu.table_id\":\"7b416700773a4e27bf152c666f48a272\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250830 01:58:28.379639 28405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.000s
@ 0x7f5419fcadcd google::LogMessage::Fail() at ??:0
@ 0x7f5419fceb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5419fca7cc google::LogMessage::Flush() at ??:0
@ 0x7f5419fcaa11 google::LogMessage::~LogMessage() at ??:0
W20250830 01:58:28.386420 28122 consensus_peers.cc:489] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 -> Peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Couldn't send request to peer 9ba697035f304124a88a1cf1aabb748e. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f54303ff794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250830 01:58:28.394086 28409 mvcc.cc:204] Tried to move back new op lower bound from 7194702251437703168 to 7194702251286728704. Current Snapshot: MvccSnapshot[applied={T|T < 7194702251437703168 or (T in {7194702251437703168})}]
I20250830 01:58:28.395762 28398 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd4f5c1e7015460e99715d8a2edd2959. Latest consensus state: current_term: 4 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:28.396047 28398 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:28.396595 28412 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:28.396934 28412 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:28.399168 28398 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:28.399370 28398 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:28.401204 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "fd4f5c1e7015460e99715d8a2edd2959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:28.401432 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:28.402478 28134 catalog_manager.cc:5869] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Sending DeleteTablet for 3 replicas of tablet cddb4c87468a44d1a2bae0fbd6784a36
@ 0x7f54303fe041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250830 01:58:28.405733 28136 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_624ff0828e534d54a8ea5f69551f7e8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:28.407053 28134 catalog_manager.cc:6652] ReplaceTablet: tablet cddb4c87468a44d1a2bae0fbd6784a36 deleted and replaced by tablet 18812772b9644792ac88671ff697e2a4
I20250830 01:58:28.407456 28134 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39492:
name: "default.Table_a36b1b6bfc854dfc907635b41d47be02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f5430405cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f5430405b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250830 01:58:28.417563 27772 tablet_service.cc:1515] Processing DeleteTablet for tablet cddb4c87468a44d1a2bae0fbd6784a36 with delete_type TABLET_DATA_DELETED (replaced by tablet 18812772b9644792ac88671ff697e2a4) from {username='slave'} at 127.0.0.1:33392
@ 0x7f541cd27cc9 kudu::Thread::SuperviseThread() at ??:0
I20250830 01:58:28.428820 27631 tablet_service.cc:1515] Processing DeleteTablet for tablet cddb4c87468a44d1a2bae0fbd6784a36 with delete_type TABLET_DATA_DELETED (replaced by tablet 18812772b9644792ac88671ff697e2a4) from {username='slave'} at 127.0.0.1:34292
I20250830 01:58:28.430285 27773 tablet_service.cc:1468] Processing CreateTablet for tablet 18812772b9644792ac88671ff697e2a4 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:28.430454 27632 tablet_service.cc:1468] Processing CreateTablet for tablet 18812772b9644792ac88671ff697e2a4 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:28.430756 27773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18812772b9644792ac88671ff697e2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:28.430799 27632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18812772b9644792ac88671ff697e2a4. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f54201496db start_thread at ??:0
@ 0x7f5418e7b71f clone at ??:0
I20250830 01:58:28.433260 28424 tablet_replica.cc:331] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: stopping tablet replica
I20250830 01:58:28.433645 28424 raft_consensus.cc:2241] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250830 01:58:28.433943 28424 raft_consensus.cc:2270] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250830 01:58:28.435181 28424 ts_tablet_manager.cc:1905] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250830 01:58:28.439881 28429 tablet_bootstrap.cc:492] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:28.440562 28424 ts_tablet_manager.cc:1918] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250830 01:58:28.440733 28424 log.cc:1199] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: Deleting WAL directory at /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal/wals/cddb4c87468a44d1a2bae0fbd6784a36
I20250830 01:58:28.441269 28424 ts_tablet_manager.cc:1939] T cddb4c87468a44d1a2bae0fbd6784a36 P 7f5563a27e4e4748813d624aab13cdca: Deleting consensus metadata
I20250830 01:58:28.442032 28429 tablet_bootstrap.cc:654] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:28.449520 28426 tablet_replica.cc:331] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: stopping tablet replica
I20250830 01:58:28.450255 28426 raft_consensus.cc:2241] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250830 01:58:28.450641 28426 raft_consensus.cc:2270] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250830 01:58:28.452502 28426 ts_tablet_manager.cc:1905] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250830 01:58:28.458734 28428 tablet_bootstrap.cc:492] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:28.460271 28428 tablet_bootstrap.cc:654] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:28.465178 28426 ts_tablet_manager.cc:1918] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250830 01:58:28.465394 28426 log.cc:1199] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: Deleting WAL directory at /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal/wals/cddb4c87468a44d1a2bae0fbd6784a36
I20250830 01:58:28.466092 28426 ts_tablet_manager.cc:1939] T cddb4c87468a44d1a2bae0fbd6784a36 P 03d61552c37a44ad9a7a6f6f5c236227: Deleting consensus metadata
I20250830 01:58:28.466567 28429 tablet_bootstrap.cc:492] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:28.466730 28429 ts_tablet_manager.cc:1397] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.013s
I20250830 01:58:28.467587 28429 raft_consensus.cc:357] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:28.467799 28429 raft_consensus.cc:383] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:28.467898 28429 raft_consensus.cc:738] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:28.468115 28429 consensus_queue.cc:260] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:28.468725 28428 tablet_bootstrap.cc:492] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:28.468854 28428 ts_tablet_manager.cc:1397] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250830 01:58:28.469487 28429 ts_tablet_manager.cc:1428] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250830 01:58:28.469729 28428 raft_consensus.cc:357] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:28.469933 28428 raft_consensus.cc:383] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:28.470019 28428 raft_consensus.cc:738] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:28.470268 28428 consensus_queue.cc:260] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:28.470700 27838 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
I20250830 01:58:28.473556 28428 ts_tablet_manager.cc:1428] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20250830 01:58:28.645853 27727 connection.cc:537] server connection from 127.0.0.1:33392 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:28.646672 28422 negotiation.cc:337] Failed RPC negotiation. Trace:
0830 01:58:28.408431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.26.105.195:35297)
0830 01:58:28.422442 (+ 14011us) server_negotiation.cc:206] Beginning negotiation
0830 01:58:28.422452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0830 01:58:28.422480 (+ 28us) server_negotiation.cc:407] Connection header received
0830 01:58:28.422566 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0830 01:58:28.422575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0830 01:58:28.422669 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0830 01:58:28.423399 (+ 730us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0830 01:58:28.423777 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0830 01:58:28.445943 (+ 22166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0830 01:58:28.646165 (+200222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44116: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":13852,"thread_start_us":13754,"threads_started":1}
W20250830 01:58:28.646689 27588 connection.cc:537] client connection to 127.26.105.252:46233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:28.647249 27728 connection.cc:537] client connection to 127.26.105.252:46233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:28.647625 27587 connection.cc:537] server connection from 127.0.0.1:34292 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:28.648108 27984 connection.cc:537] client connection to 127.26.105.252:46233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:28.648720 27984 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.26.105.252:46233)
W20250830 01:58:28.649271 27984 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.26.105.252:46233)
W20250830 01:58:28.649537 27984 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.105.252:46233)
I20250830 01:58:28.652292 28272 raft_consensus.cc:491] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:28.652462 28272 raft_consensus.cc:513] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:28.653088 28272 leader_election.cc:290] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297), 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:28.653615 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18812772b9644792ac88671ff697e2a4" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:28.653841 27652 raft_consensus.cc:2466] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
W20250830 01:58:28.653878 27729 leader_election.cc:343] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297): Not found: Tablet not found: 18812772b9644792ac88671ff697e2a4
I20250830 01:58:28.654242 27727 leader_election.cc:304] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters: 893b0b641a51432f9172ed0b41521822
I20250830 01:58:28.654480 28272 raft_consensus.cc:2802] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:28.654603 28272 raft_consensus.cc:491] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:28.654687 28272 raft_consensus.cc:3058] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:28.657644 28272 raft_consensus.cc:513] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:28.658206 28272 leader_election.cc:290] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297), 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:28.658799 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18812772b9644792ac88671ff697e2a4" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:28.659029 27652 raft_consensus.cc:3058] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
W20250830 01:58:28.659093 27729 leader_election.cc:343] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297): Not found: Tablet not found: 18812772b9644792ac88671ff697e2a4
I20250830 01:58:28.661365 27652 raft_consensus.cc:2466] T 18812772b9644792ac88671ff697e2a4 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:28.661767 27727 leader_election.cc:304] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters: 893b0b641a51432f9172ed0b41521822
I20250830 01:58:28.662065 28272 raft_consensus.cc:2802] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:28.662454 28272 raft_consensus.cc:695] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:28.662792 28272 consensus_queue.cc:237] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:28.906527 27974 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
I20250830 01:58:28.925117 28443 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fd4f5c1e7015460e99715d8a2edd2959)
I20250830 01:58:28.925309 28443 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:28.926045 28443 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:28.928054 28319 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:28.928223 28319 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:28.928303 28319 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:28.928532 28443 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250830 01:58:28.994305 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 1: this message will repeat every 5th retry.
W20250830 01:58:29.057261 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 1: this message will repeat every 5th retry.
I20250830 01:58:29.502333 28450 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fd4f5c1e7015460e99715d8a2edd2959)
I20250830 01:58:29.502542 28450 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:29.503284 28450 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:29.505492 28319 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:29.505664 28319 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:29.505805 28319 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:29.506062 28450 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250830 01:58:29.623237 28272 consensus_queue.cc:1035] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250830 01:58:29.629212 27729 consensus_peers.cc:489] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca -> Peer 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297): Couldn't send request to peer 893b0b641a51432f9172ed0b41521822. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18812772b9644792ac88671ff697e2a4. This is attempt 1: this message will repeat every 5th retry.
I20250830 01:58:30.138238 27047 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu pid 28295
I20250830 01:58:30.144590 28363 generic_service.cc:196] Checking for leaks (request via RPC)
I20250830 01:58:30.260841 27047 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu with pid 28295
I20250830 01:58:30.715293 28475 ts_tablet_manager.cc:927] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: Initiating tablet copy from peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:30.716077 28475 tablet_copy_client.cc:323] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: tablet copy: Beginning tablet copy session from remote peer at address 127.26.105.194:43787
I20250830 01:58:30.717015 27813 tablet_copy_service.cc:140] P 7f5563a27e4e4748813d624aab13cdca: Received BeginTabletCopySession request for tablet 18812772b9644792ac88671ff697e2a4 from peer 893b0b641a51432f9172ed0b41521822 ({username='slave'} at 127.26.105.195:52889)
I20250830 01:58:30.717177 27813 tablet_copy_service.cc:161] P 7f5563a27e4e4748813d624aab13cdca: Beginning new tablet copy session on tablet 18812772b9644792ac88671ff697e2a4 from peer 893b0b641a51432f9172ed0b41521822 at {username='slave'} at 127.26.105.195:52889: session id = 893b0b641a51432f9172ed0b41521822-18812772b9644792ac88671ff697e2a4
I20250830 01:58:30.719635 27813 tablet_copy_source_session.cc:215] T 18812772b9644792ac88671ff697e2a4 P 7f5563a27e4e4748813d624aab13cdca: Tablet Copy: opened 0 blocks and 1 log segments
I20250830 01:58:30.721451 28475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18812772b9644792ac88671ff697e2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:30.725912 28475 tablet_copy_client.cc:806] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: tablet copy: Starting download of 0 data blocks...
I20250830 01:58:30.726230 28475 tablet_copy_client.cc:670] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: tablet copy: Starting download of 1 WAL segments...
I20250830 01:58:30.728724 28475 tablet_copy_client.cc:538] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250830 01:58:30.731592 28475 tablet_bootstrap.cc:492] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:30.736630 28475 tablet_bootstrap.cc:492] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250830 01:58:30.737063 28475 tablet_bootstrap.cc:492] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: Bootstrap complete.
I20250830 01:58:30.737334 28475 ts_tablet_manager.cc:1397] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250830 01:58:30.737946 28475 raft_consensus.cc:357] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:30.738101 28475 raft_consensus.cc:738] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:30.738350 28475 consensus_queue.cc:260] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } }
I20250830 01:58:30.739095 28475 ts_tablet_manager.cc:1428] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250830 01:58:30.740742 27813 tablet_copy_service.cc:342] P 7f5563a27e4e4748813d624aab13cdca: Request end of tablet copy session 893b0b641a51432f9172ed0b41521822-18812772b9644792ac88671ff697e2a4 received from {username='slave'} at 127.26.105.195:52889
I20250830 01:58:30.740911 27813 tablet_copy_service.cc:434] P 7f5563a27e4e4748813d624aab13cdca: ending tablet copy session 893b0b641a51432f9172ed0b41521822-18812772b9644792ac88671ff697e2a4 on tablet 18812772b9644792ac88671ff697e2a4 with peer 893b0b641a51432f9172ed0b41521822
I20250830 01:58:30.823958 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--webserver_interface=127.26.105.254
--webserver_port=37963
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:31.166599 28481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:31.166925 28481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:31.167047 28481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:31.176234 28481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:31.176335 28481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:31.176400 28481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:31.176467 28481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:31.186501 28481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.105.254
--webserver_port=37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:31.188135 28481 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:31.189918 28481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:31.197875 28491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:31.199643 28492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:31.200127 28494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:31.200903 28481 server_base.cc:1047] running on GCE node
I20250830 01:58:31.201936 28481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:31.203517 28481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:31.204728 28481 hybrid_clock.cc:648] HybridClock initialized: now 1756519111204667 us; error 56 us; skew 500 ppm
I20250830 01:58:31.205078 28481 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:31.207316 28481 webserver.cc:480] Webserver started at http://127.26.105.254:37963/ using document root <none> and password file <none>
I20250830 01:58:31.208124 28481 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:31.208233 28481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:31.213624 28481 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250830 01:58:31.216589 28500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:31.217996 28481 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250830 01:58:31.218218 28481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
uuid: "ad9824f7e6604077ac8a034879b4c731"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:31.219115 28481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:31.237990 28481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:31.238930 28481 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:31.239218 28481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:31.260030 28481 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.254:36635
I20250830 01:58:31.260077 28551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.254:36635 every 8 connection(s)
I20250830 01:58:31.261677 28481 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
I20250830 01:58:31.265571 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 28481
I20250830 01:58:31.269055 28552 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:31.272627 28552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap starting.
I20250830 01:58:31.287002 28552 log.cc:826] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:31.306720 28552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=22 ignored=0} inserts{seen=15 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250830 01:58:31.307314 28552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap complete.
I20250830 01:58:31.312508 28552 raft_consensus.cc:357] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:31.315187 28552 raft_consensus.cc:738] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Initialized, Role: FOLLOWER
I20250830 01:58:31.315914 28552 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 4.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:31.316622 28558 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:31.316885 28558 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:31.317633 28552 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:31.331739 28572 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:31.331916 28572 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: loading cluster ID for follower catalog manager: success
I20250830 01:58:31.336419 28572 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: acquiring CA information for follower catalog manager: success
I20250830 01:58:31.339483 28572 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:31.347996 28481 master_runner.cc:186] Error getting master registration for 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111),
I20250830 01:58:31.354429 28481 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:31.657150 27700 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:31.660490 28517 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:37039; Asking this server to re-register.
I20250830 01:58:31.660948 27839 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:31.662302 28517 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:57521; Asking this server to re-register.
I20250830 01:58:31.662397 27700 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:31.662901 27839 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:31.664381 28517 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:31.664503 28516 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:31.680234 27975 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:31.682036 28517 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:51259; Asking this server to re-register.
I20250830 01:58:31.682715 27975 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:31.684022 28517 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:31.796180 27929 raft_consensus.cc:1215] T 18812772b9644792ac88671ff697e2a4 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250830 01:58:31.864701 28587 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:31.864938 28587 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:31.866586 28587 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:31.867841 28503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:31.869225 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:31.869496 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:31.869602 28503 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:31.869959 28587 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250830 01:58:32.468283 28599 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:32.468503 28599 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:32.469187 28599 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:32.473762 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:32.473973 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:32.474104 28503 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:32.474411 28599 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250830 01:58:33.320032 28608 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:33.320238 28608 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:33.321033 28608 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:33.325724 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:33.326119 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:33.326426 28503 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:33.326711 28608 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250830 01:58:34.592024 28627 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:34.592197 28627 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:34.592847 28627 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:34.595486 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:34.595682 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:34.595793 28503 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:34.596056 28627 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250830 01:58:34.601995 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.253:39521
--webserver_interface=127.26.105.253
--webserver_port=37387
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:34.939932 28631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:34.940306 28631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:34.940448 28631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:34.948971 28631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:34.949079 28631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:34.949136 28631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:34.949204 28631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:34.959427 28631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.253:39521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.105.253
--webserver_port=37387
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:34.961055 28631 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:34.962939 28631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:34.971096 28642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:34.971174 28643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:34.971153 28645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:34.972265 28631 server_base.cc:1047] running on GCE node
I20250830 01:58:34.973055 28631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:34.974560 28631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:34.975776 28631 hybrid_clock.cc:648] HybridClock initialized: now 1756519114975725 us; error 42 us; skew 500 ppm
I20250830 01:58:34.976111 28631 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:34.978374 28631 webserver.cc:480] Webserver started at http://127.26.105.253:37387/ using document root <none> and password file <none>
I20250830 01:58:34.979133 28631 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:34.979264 28631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:34.984508 28631 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20250830 01:58:34.987362 28651 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:34.988734 28631 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250830 01:58:34.988905 28631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
uuid: "9ba697035f304124a88a1cf1aabb748e"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:34.989594 28631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:35.051106 28631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:35.051963 28631 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:35.052284 28631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:35.072832 28631 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.253:39521
I20250830 01:58:35.072856 28702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.253:39521 every 8 connection(s)
I20250830 01:58:35.074599 28631 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
I20250830 01:58:35.075170 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 28631
I20250830 01:58:35.083348 28703 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:35.087718 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Bootstrap starting.
I20250830 01:58:35.097663 28703 log.cc:826] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:35.117303 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=19 ignored=0} inserts{seen=14 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250830 01:58:35.117918 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Bootstrap complete.
I20250830 01:58:35.121477 27976 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:35.124768 28703 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:35.127728 28703 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba697035f304124a88a1cf1aabb748e, State: Initialized, Role: FOLLOWER
I20250830 01:58:35.128569 28703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 3.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:35.129360 28708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:35.129608 28708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:35.130441 28703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:35.144371 28725 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:35.144591 28725 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: loading cluster ID for follower catalog manager: success
I20250830 01:58:35.148918 28725 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: acquiring CA information for follower catalog manager: success
I20250830 01:58:35.153699 28725 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:35.182931 27701 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:35.182337 28631 master_runner.cc:186] Error getting master registration for 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111),
I20250830 01:58:35.188016 28631 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:35.188637 28668 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:45369; Asking this server to re-register.
I20250830 01:58:35.189493 27701 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:35.191183 28668 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:35.315524 27842 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:35.317631 28668 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:44751; Asking this server to re-register.
I20250830 01:58:35.318384 27842 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:35.319804 28668 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:35.621359 28708 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:35.621625 28708 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:35.623483 28708 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:35.624271 28527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
W20250830 01:58:35.624328 28654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:35.624727 28527 raft_consensus.cc:2391] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ba697035f304124a88a1cf1aabb748e in current term 4: Already voted for candidate fd4f5c1e7015460e99715d8a2edd2959 in this term.
W20250830 01:58:35.625838 28654 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:35.625967 28654 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:35.626360 28708 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 3 FOLLOWER]: Advancing to term 4
I20250830 01:58:35.629671 28708 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250830 01:58:35.637923 28668 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:46765; Asking this server to re-register.
I20250830 01:58:35.638621 27976 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:35.639752 28668 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:36.254910 28747 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:36.255165 28747 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:36.255909 28747 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:36.256496 28527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 5 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:36.256771 28527 raft_consensus.cc:2408] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ba697035f304124a88a1cf1aabb748e for term 5 because replica has last-logged OpId of term: 4 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 20.
W20250830 01:58:36.258011 28654 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:36.258134 28654 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:36.258437 28747 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250830 01:58:36.366523 28481 master_runner.cc:186] Error getting master registration for 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111),
I20250830 01:58:36.366753 28481 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:36.878571 28760 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:36.878744 28760 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:36.879364 28760 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:36.879913 28527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 5 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:36.880163 28527 raft_consensus.cc:2408] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ba697035f304124a88a1cf1aabb748e for term 5 because replica has last-logged OpId of term: 4 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 20.
W20250830 01:58:36.881359 28654 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:36.881492 28654 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:36.881728 28760 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250830 01:58:37.069834 28763 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:37.070026 28763 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:37.070813 28763 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:37.071700 28678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e" is_pre_election: true
W20250830 01:58:37.072095 28503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20250830 01:58:37.072369 28678 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 4.
I20250830 01:58:37.073107 28503 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters:
I20250830 01:58:37.073418 28763 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250830 01:58:37.073563 28763 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:37.073727 28763 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 4 FOLLOWER]: Advancing to term 5
W20250830 01:58:37.073927 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:37.078387 28763 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:37.079221 28763 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 election: Requested vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:37.079756 28678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e"
I20250830 01:58:37.079959 28678 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 4 FOLLOWER]: Advancing to term 5
W20250830 01:58:37.080878 28503 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:37.082393 28678 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 5.
I20250830 01:58:37.082834 28503 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters: fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:37.083163 28763 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Leader election won for term 5
I20250830 01:58:37.083602 28763 raft_consensus.cc:695] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 LEADER]: Becoming Leader. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Running, Role: LEADER
I20250830 01:58:37.084126 28763 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.23, Last appended by leader: 23, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:37.086746 28766 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 5 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:37.087031 28766 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:37.182952 27047 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu pid 28631
I20250830 01:58:37.190519 28698 generic_service.cc:196] Checking for leaks (request via RPC)
I20250830 01:58:37.300024 27047 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu with pid 28631
W20250830 01:58:37.625679 28503 consensus_peers.cc:489] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 -> Peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Couldn't send request to peer fd4f5c1e7015460e99715d8a2edd2959. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250830 01:58:37.691548 28503 consensus_peers.cc:489] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 -> Peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Couldn't send request to peer 9ba697035f304124a88a1cf1aabb748e. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250830 01:58:37.950793 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.253:39521
--webserver_interface=127.26.105.253
--webserver_port=37387
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:38.094709 28768 op_tracker.cc:284] OpTracker waiting for 1 outstanding ops to complete now for 1.007s
I20250830 01:58:38.094869 28768 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20250830 01:58:38.094944 28768 op_tracker.cc:290] R-P WriteOp [type=FOLLOWER, start_time=2025-08-30 01:58:31, state=WriteOpState 0x61700002ff00 [op_id=(term: 4 index: 23), ts=7194702268053753856, rows=[MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>]]]
W20250830 01:58:38.290148 28790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:38.290499 28790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:38.290586 28790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:38.299117 28790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:38.299212 28790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:38.299294 28790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:38.299381 28790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:38.310106 28790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.253:39521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.105.253
--webserver_port=37387
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:38.311810 28790 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:38.313553 28790 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:38.321385 28795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:38.322856 28796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:38.323470 28798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:38.324364 28790 server_base.cc:1047] running on GCE node
I20250830 01:58:38.325325 28790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:38.326962 28790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:38.328191 28790 hybrid_clock.cc:648] HybridClock initialized: now 1756519118328145 us; error 42 us; skew 500 ppm
I20250830 01:58:38.328573 28790 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:38.331229 28790 webserver.cc:480] Webserver started at http://127.26.105.253:37387/ using document root <none> and password file <none>
I20250830 01:58:38.331988 28790 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:38.332094 28790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:38.337656 28790 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20250830 01:58:38.340646 28804 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:38.342078 28790 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250830 01:58:38.342298 28790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
uuid: "9ba697035f304124a88a1cf1aabb748e"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:38.343014 28790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:38.358027 28790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:38.358927 28790 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:38.359212 28790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:38.378443 28790 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.253:39521
I20250830 01:58:38.378461 28855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.253:39521 every 8 connection(s)
I20250830 01:58:38.380175 28790 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-1/data/info.pb
I20250830 01:58:38.380856 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 28790
I20250830 01:58:38.388262 28856 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:38.393494 28856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Bootstrap starting.
I20250830 01:58:38.408844 27976 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:38.408950 28856 log.cc:826] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:38.436877 28856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=19 ignored=0} inserts{seen=14 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250830 01:58:38.437856 28856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Bootstrap complete.
I20250830 01:58:38.443822 28856 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:38.446672 28856 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba697035f304124a88a1cf1aabb748e, State: Initialized, Role: FOLLOWER
I20250830 01:58:38.447417 28856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 3.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:38.448222 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:38.448504 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:38.449395 28856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:38.467204 28882 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:38.467414 28882 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: loading cluster ID for follower catalog manager: success
I20250830 01:58:38.471750 28882 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: acquiring CA information for follower catalog manager: success
I20250830 01:58:38.474998 28882 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:38.490830 28790 master_runner.cc:186] Error getting master registration for 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111),
I20250830 01:58:38.494406 28790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:58:38.776278 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 6: this message will repeat every 5th retry.
I20250830 01:58:38.777242 28820 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:54145; Asking this server to re-register.
I20250830 01:58:38.778137 27976 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:38.780117 28820 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:38.829288 27701 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:38.830646 28820 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:60699; Asking this server to re-register.
I20250830 01:58:38.831980 27701 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:38.833997 28821 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:38.837743 27842 heartbeater.cc:344] Connected to a master server at 127.26.105.253:39521
I20250830 01:58:38.838963 28821 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:37493; Asking this server to re-register.
I20250830 01:58:38.839600 27842 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:38.840829 28821 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:38.856673 28831 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Refusing update from remote peer ad9824f7e6604077ac8a034879b4c731: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 4 index: 23. (index mismatch)
I20250830 01:58:38.857681 28780 consensus_queue.cc:1037] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 19, Time since last communication: 0.000s
I20250830 01:58:38.863274 28890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:38.863788 28867 mvcc.cc:204] Tried to move back new op lower bound from 7194702267903205376 to 7194702267781169152. Current Snapshot: MvccSnapshot[applied={T|T < 7194702267903205376 or (T in {7194702267903205376})}]
I20250830 01:58:38.863909 28890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:38.864074 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 5 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:38.864286 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:38.866286 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:38.866544 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:38.868039 28780 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:38.868216 28780 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:39.052714 28768 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:39.055209 28768 catalog_manager.cc:671] Loaded metadata for table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]
I20250830 01:58:39.055621 28768 catalog_manager.cc:671] Loaded metadata for table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]
I20250830 01:58:39.055932 28768 catalog_manager.cc:671] Loaded metadata for table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]
I20250830 01:58:39.056197 28768 catalog_manager.cc:671] Loaded metadata for table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]
I20250830 01:58:39.057943 28768 tablet_loader.cc:96] loaded metadata for tablet 18812772b9644792ac88671ff697e2a4 (table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a])
I20250830 01:58:39.058454 28768 tablet_loader.cc:96] loaded metadata for tablet 47a4f27ca16d436386ed86810020f84b (table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f])
I20250830 01:58:39.058714 28768 tablet_loader.cc:96] loaded metadata for tablet 779c87f5e2c34e8abd116c7c1c4f0092 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368])
I20250830 01:58:39.058936 28768 tablet_loader.cc:96] loaded metadata for tablet 876295e43d7c4b01b2f772a492bc27cf (table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46])
I20250830 01:58:39.059468 28768 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:39.060303 28768 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:39.060393 28768 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:39.061501 28768 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:39.062410 28768 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Loaded TSK: 0
I20250830 01:58:39.063206 28768 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:39.064045 28768 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:39.064826 28768 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250830 01:58:39.119542 28516 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:53528
I20250830 01:58:39.120054 28514 master_service.cc:948] ReplaceTablet: received request to replace tablet 3996bdda030340d081af363dbdfea328 from {username='slave'} at 127.0.0.1:53528
I20250830 01:58:39.121057 28517 master_service.cc:948] ReplaceTablet: received request to replace tablet 779c87f5e2c34e8abd116c7c1c4f0092 from {username='slave'} at 127.0.0.1:53528
I20250830 01:58:39.121161 28512 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53528:
table { table_name: "default.Table_2c93d305666e48d5b919f93ebe9e05ec" } new_table_name: "default.Table_942f583f6001419a91d24f8fdb74108c" modify_external_catalogs: true
I20250830 01:58:39.121311 28516 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53528:
name: "default.Table_509af6139e2e4ba6bc5c537af315df6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:39.121886 28510 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53528:
name: "default.Table_d4634c7907fc46558619aee806e21da3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:39.122869 28514 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53528:
name: "default.Table_ac070120b97141e188efab1e24ece67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:39.123750 28515 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:53528
I20250830 01:58:39.124027 28511 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53528:
table { table_name: "default.Table_2c4fc026b80045d8a58497a870104278" } new_table_name: "default.Table_f8d128ad097b4e25a4cc03e5b2f02676" modify_external_catalogs: true
I20250830 01:58:39.124421 28513 master_service.cc:948] ReplaceTablet: received request to replace tablet 779c87f5e2c34e8abd116c7c1c4f0092 from {username='slave'} at 127.0.0.1:53528
I20250830 01:58:39.124593 28509 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53528:
name: "default.Table_11641b1a96984fa099ffa232b6a0033e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:39.125358 28515 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53528:
name: "default.Table_badff657ae07402a88632f78ec1befb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:39.125494 28508 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53528:
name: "default.Table_1ad78e45bc5842aeaf69a74b3ef38007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250830 01:58:39.127456 28571 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 5 received events: NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_488d167c0efe4d1b91a27f563856f9f6, message={"server":"","servicePrincipal":"","db":"default","table":"table_488d167c0efe4d1b91a27f563856f9f6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_488d167c0efe4d1b91a27f563856f9f6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_488d167c0efe4d1b91a27f563856f9f6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_488d167c0efe4d1b91a27f563856f9f6\",\"kudu.table_id\":\"a66ba6403ba349f3ade7628ab65cd30f\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_badff657ae07402a88632f78ec1befb7, message={"server":"","servicePrincipal":"","db":"default","table":"table_badff657ae07402a88632f78ec1befb7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_badff657ae07402a88632f78ec1befb7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_badff657ae07402a88632f78ec1befb7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_badff657ae07402a88632f78ec1befb7\",\"kudu.table_id\":\"7b416700773a4e27bf152c666f48a272\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f7326cbadcd google::LogMessage::Fail() at ??:0
@ 0x7f7326cbeb93 google::LogMessage::SendToLog() at ??:0
I20250830 01:58:39.132014 28517 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Sending DeleteTablet for 3 replicas of tablet 779c87f5e2c34e8abd116c7c1c4f0092
I20250830 01:58:39.132565 28513 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53528:
table { table_name: "default.Table_d19bb3dab12e478cbd8897446f84d508" } new_table_name: "default.Table_31e5e45019fd41888d6a471fbd3736b4" modify_external_catalogs: true
@ 0x7f7326cba7cc google::LogMessage::Flush() at ??:0
I20250830 01:58:39.134050 28517 catalog_manager.cc:6652] ReplaceTablet: tablet 779c87f5e2c34e8abd116c7c1c4f0092 deleted and replaced by tablet 7f45870df10d4910a749167310afb2e5
I20250830 01:58:39.134462 28517 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53528:
name: "default.Table_739ceee58b144d72b56342855499ecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f7326cbaa11 google::LogMessage::~LogMessage() at ??:0
I20250830 01:58:39.150784 27908 tablet_service.cc:1515] Processing DeleteTablet for tablet 779c87f5e2c34e8abd116c7c1c4f0092 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f45870df10d4910a749167310afb2e5) from {username='slave'} at 127.0.0.1:53070
I20250830 01:58:39.151211 27631 tablet_service.cc:1515] Processing DeleteTablet for tablet 779c87f5e2c34e8abd116c7c1c4f0092 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f45870df10d4910a749167310afb2e5) from {username='slave'} at 127.0.0.1:42994
I20250830 01:58:39.152617 28905 tablet_replica.cc:331] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: stopping tablet replica
I20250830 01:58:39.152832 28905 raft_consensus.cc:2241] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250830 01:58:39.152961 28904 tablet_replica.cc:331] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: stopping tablet replica
I20250830 01:58:39.153079 28905 raft_consensus.cc:2270] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250830 01:58:39.153420 28904 raft_consensus.cc:2241] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250830 01:58:39.153671 28905 ts_tablet_manager.cc:1905] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250830 01:58:39.153700 28904 raft_consensus.cc:2270] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250830 01:58:39.154780 28904 ts_tablet_manager.cc:1905] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250830 01:58:39.160285 28905 ts_tablet_manager.cc:1918] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250830 01:58:39.160442 28905 log.cc:1199] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: Deleting WAL directory at /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-0/wal/wals/779c87f5e2c34e8abd116c7c1c4f0092
I20250830 01:58:39.160851 28905 ts_tablet_manager.cc:1939] T 779c87f5e2c34e8abd116c7c1c4f0092 P 03d61552c37a44ad9a7a6f6f5c236227: Deleting consensus metadata
I20250830 01:58:39.161216 28904 ts_tablet_manager.cc:1918] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250830 01:58:39.161366 28904 log.cc:1199] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: Deleting WAL directory at /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-2/wal/wals/779c87f5e2c34e8abd116c7c1c4f0092
I20250830 01:58:39.161835 28904 ts_tablet_manager.cc:1939] T 779c87f5e2c34e8abd116c7c1c4f0092 P 893b0b641a51432f9172ed0b41521822: Deleting consensus metadata
I20250830 01:58:39.166738 28502 catalog_manager.cc:4928] TS 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): tablet 779c87f5e2c34e8abd116c7c1c4f0092 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]) successfully deleted
I20250830 01:58:39.167441 27773 tablet_service.cc:1515] Processing DeleteTablet for tablet 779c87f5e2c34e8abd116c7c1c4f0092 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f45870df10d4910a749167310afb2e5) from {username='slave'} at 127.0.0.1:34348
I20250830 01:58:39.167670 28504 catalog_manager.cc:4928] TS 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297): tablet 779c87f5e2c34e8abd116c7c1c4f0092 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]) successfully deleted
I20250830 01:58:39.168810 28907 tablet_replica.cc:331] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: stopping tablet replica
@ 0x7f733d0ef794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250830 01:58:39.169198 28907 raft_consensus.cc:2241] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Raft consensus shutting down.
I20250830 01:58:39.170086 28907 raft_consensus.cc:2270] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250830 01:58:39.170858 28907 ts_tablet_manager.cc:1905] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f733d0ee041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250830 01:58:39.179431 28907 ts_tablet_manager.cc:1918] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250830 01:58:39.179581 28907 log.cc:1199] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: Deleting WAL directory at /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/ts-1/wal/wals/779c87f5e2c34e8abd116c7c1c4f0092
I20250830 01:58:39.180083 28907 ts_tablet_manager.cc:1939] T 779c87f5e2c34e8abd116c7c1c4f0092 P 7f5563a27e4e4748813d624aab13cdca: Deleting consensus metadata
@ 0x7f733d0f5cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250830 01:58:39.182062 28503 catalog_manager.cc:4928] TS 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): tablet 779c87f5e2c34e8abd116c7c1c4f0092 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]) successfully deleted
@ 0x7f733d0f5b39 std::_Function_handler<>::_M_invoke() at ??:0
I20250830 01:58:39.185330 27773 tablet_service.cc:1468] Processing CreateTablet for tablet 7f45870df10d4910a749167310afb2e5 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:39.185860 27773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f45870df10d4910a749167310afb2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:39.185847 27631 tablet_service.cc:1468] Processing CreateTablet for tablet 7f45870df10d4910a749167310afb2e5 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:39.186297 27631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f45870df10d4910a749167310afb2e5. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0xa774dd std::function<>::operator()() at ??:?
I20250830 01:58:39.190765 27908 tablet_service.cc:1468] Processing CreateTablet for tablet 7f45870df10d4910a749167310afb2e5 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f7329a17cc9 kudu::Thread::SuperviseThread() at ??:0
I20250830 01:58:39.191195 27908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f45870df10d4910a749167310afb2e5. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f732ce396db start_thread at ??:0
@ 0x7f7325b6b71f clone at ??:0
I20250830 01:58:39.207307 28914 tablet_bootstrap.cc:492] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:39.208467 28913 tablet_bootstrap.cc:492] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:39.209713 28914 tablet_bootstrap.cc:654] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:39.211048 28913 tablet_bootstrap.cc:654] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:39.214732 28914 tablet_bootstrap.cc:492] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:39.214921 28914 ts_tablet_manager.cc:1397] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250830 01:58:39.216020 28914 raft_consensus.cc:357] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.216277 28914 raft_consensus.cc:383] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:39.216378 28914 raft_consensus.cc:738] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:39.216672 28914 consensus_queue.cc:260] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.218892 28915 tablet_bootstrap.cc:492] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:39.219059 28914 ts_tablet_manager.cc:1428] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250830 01:58:39.220912 28915 tablet_bootstrap.cc:654] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:39.223037 28913 tablet_bootstrap.cc:492] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:39.223193 28913 ts_tablet_manager.cc:1397] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20250830 01:58:39.224095 28913 raft_consensus.cc:357] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.224295 28913 raft_consensus.cc:383] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:39.224388 28913 raft_consensus.cc:738] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:39.224651 28913 consensus_queue.cc:260] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.225244 28913 ts_tablet_manager.cc:1428] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250830 01:58:39.228446 28915 tablet_bootstrap.cc:492] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:39.228605 28915 ts_tablet_manager.cc:1397] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250830 01:58:39.229519 28915 raft_consensus.cc:357] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.229704 28915 raft_consensus.cc:383] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:39.229799 28915 raft_consensus.cc:738] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:39.230033 28915 consensus_queue.cc:260] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.230672 28915 ts_tablet_manager.cc:1428] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250830 01:58:39.343072 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 6: this message will repeat every 5th retry.
W20250830 01:58:39.389614 27728 connection.cc:537] server connection from 127.0.0.1:34348 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:39.389708 27862 connection.cc:537] server connection from 127.0.0.1:53070 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:39.389699 27586 connection.cc:537] server connection from 127.0.0.1:42994 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:39.390075 27865 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:39.390479 27589 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:39.390635 27985 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:39.391681 27985 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.26.105.254:36635)
W20250830 01:58:39.391626 27729 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:39.392302 27985 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.26.105.254:36635)
W20250830 01:58:39.392539 27985 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.105.254:36635)
W20250830 01:58:39.394516 28783 negotiation.cc:337] Failed RPC negotiation. Trace:
0830 01:58:39.392913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.105.254:36635 (local address 127.26.105.193:55333)
0830 01:58:39.393225 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0830 01:58:39.393242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0830 01:58:39.393343 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0830 01:58:39.394293 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":211}
W20250830 01:58:39.395010 28860 negotiation.cc:337] Failed RPC negotiation. Trace:
0830 01:58:39.391009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.105.254:36635 (local address 127.26.105.195:56395)
0830 01:58:39.392373 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0830 01:58:39.392393 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0830 01:58:39.392521 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0830 01:58:39.394816 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1242}
I20250830 01:58:39.684335 28927 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:39.684609 28927 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:39.686270 28927 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:39.687270 28807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:39.688480 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:39.688707 28808 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:39.688828 28808 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:39.689127 28927 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250830 01:58:39.700928 28887 raft_consensus.cc:491] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:39.701095 28887 raft_consensus.cc:513] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.701866 28887 leader_election.cc:290] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:39.702420 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f45870df10d4910a749167310afb2e5" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:39.702450 27793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f45870df10d4910a749167310afb2e5" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5563a27e4e4748813d624aab13cdca" is_pre_election: true
I20250830 01:58:39.702664 27929 raft_consensus.cc:2466] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 0.
I20250830 01:58:39.702657 27793 raft_consensus.cc:2466] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 0.
I20250830 01:58:39.703199 27589 leader_election.cc:304] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters:
I20250830 01:58:39.703495 28887 raft_consensus.cc:2802] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:39.703598 28887 raft_consensus.cc:491] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:39.703691 28887 raft_consensus.cc:3058] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:39.705523 28887 raft_consensus.cc:513] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:39.706087 28887 leader_election.cc:290] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 election: Requested vote from peers 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:39.706800 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f45870df10d4910a749167310afb2e5" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:39.706800 27793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f45870df10d4910a749167310afb2e5" candidate_uuid: "03d61552c37a44ad9a7a6f6f5c236227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5563a27e4e4748813d624aab13cdca"
I20250830 01:58:39.707036 27793 raft_consensus.cc:3058] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:39.707028 27929 raft_consensus.cc:3058] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:39.709839 27793 raft_consensus.cc:2466] T 7f45870df10d4910a749167310afb2e5 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 1.
I20250830 01:58:39.709839 27929 raft_consensus.cc:2466] T 7f45870df10d4910a749167310afb2e5 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03d61552c37a44ad9a7a6f6f5c236227 in term 1.
I20250830 01:58:39.710333 27588 leader_election.cc:304] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters:
I20250830 01:58:39.710629 28887 raft_consensus.cc:2802] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:39.711050 28887 raft_consensus.cc:695] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 LEADER]: Becoming Leader. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Running, Role: LEADER
I20250830 01:58:39.711431 28887 consensus_queue.cc:237] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:40.371183 28935 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:40.371363 28935 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:40.372028 28935 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:40.374089 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:40.374332 28808 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:40.374470 28808 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:40.374761 28935 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250830 01:58:40.606881 28887 consensus_queue.cc:1035] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:40.614562 28948 consensus_queue.cc:1035] T 7f45870df10d4910a749167310afb2e5 P 03d61552c37a44ad9a7a6f6f5c236227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:41.313596 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--webserver_interface=127.26.105.254
--webserver_port=37963
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250830 01:58:41.483762 28962 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:41.483966 28962 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:41.484593 28962 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:41.486929 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
W20250830 01:58:41.487133 28808 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:41.487232 28808 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:41.487529 28962 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250830 01:58:41.673542 28958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:41.673884 28958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:41.673974 28958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:41.682384 28958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:41.682485 28958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:41.682545 28958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:41.682601 28958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:41.692435 28958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.105.254
--webserver_port=37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:41.694082 28958 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:41.695896 28958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:41.704988 28974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:41.705497 28975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:41.705968 28977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:41.706984 28958 server_base.cc:1047] running on GCE node
I20250830 01:58:41.707789 28958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:41.709437 28958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:41.710754 28958 hybrid_clock.cc:648] HybridClock initialized: now 1756519121710662 us; error 67 us; skew 500 ppm
I20250830 01:58:41.711180 28958 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:41.714462 28958 webserver.cc:480] Webserver started at http://127.26.105.254:37963/ using document root <none> and password file <none>
I20250830 01:58:41.715286 28958 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:41.715385 28958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:41.721074 28958 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250830 01:58:41.724501 28983 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:41.725965 28958 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250830 01:58:41.726143 28958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
uuid: "ad9824f7e6604077ac8a034879b4c731"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:41.726920 28958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:41.742635 28958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:41.743525 28958 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:41.743887 28958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:41.766306 28958 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.254:36635
I20250830 01:58:41.766350 29034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.254:36635 every 8 connection(s)
I20250830 01:58:41.767936 28958 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
I20250830 01:58:41.768232 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 28958
I20250830 01:58:41.775593 29035 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:41.780174 29035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap starting.
I20250830 01:58:41.791174 29035 log.cc:826] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:41.815186 29035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=16 ignored=0} mutations{seen=14 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250830 01:58:41.815825 29035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap complete.
I20250830 01:58:41.821271 29035 raft_consensus.cc:357] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:41.821961 29035 raft_consensus.cc:738] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Initialized, Role: FOLLOWER
I20250830 01:58:41.822636 29035 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 5.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:41.823737 29039 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:41.824098 29039 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:41.825275 29035 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:41.842140 29053 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:41.842379 29053 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: loading cluster ID for follower catalog manager: success
I20250830 01:58:41.846753 29053 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: acquiring CA information for follower catalog manager: success
I20250830 01:58:41.850248 29053 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:41.861886 28958 master_runner.cc:186] Error getting master registration for 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111),
I20250830 01:58:41.867946 28958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:42.035586 27700 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:42.040599 29000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:46857; Asking this server to re-register.
I20250830 01:58:42.042615 27700 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:42.045758 28999 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:42.052296 27975 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:42.054284 29000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:41747; Asking this server to re-register.
I20250830 01:58:42.055079 27975 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:42.056381 29000 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:42.135589 27839 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:42.137125 29000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:52013; Asking this server to re-register.
I20250830 01:58:42.137709 27839 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:42.139011 29000 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:42.252920 29039 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:42.253165 29039 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:42.255043 29039 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:42.255645 28831 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 6 candidate_status { last_received { term: 5 index: 26 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e" is_pre_election: true
W20250830 01:58:42.255792 28986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:42.256002 28831 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 5.
I20250830 01:58:42.256747 28986 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters:
I20250830 01:58:42.257161 29039 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250830 01:58:42.257300 29039 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:42.257414 29039 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 5 FOLLOWER]: Advancing to term 6
W20250830 01:58:42.257620 28986 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:42.262084 29039 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:42.262934 29039 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 6 election: Requested vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:42.263414 28831 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 6 candidate_status { last_received { term: 5 index: 26 } } ignore_live_leader: false dest_uuid: "9ba697035f304124a88a1cf1aabb748e"
I20250830 01:58:42.263653 28831 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 5 FOLLOWER]: Advancing to term 6
W20250830 01:58:42.264322 28986 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:42.266957 28831 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 6.
I20250830 01:58:42.267419 28986 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters: fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:42.267792 29039 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 6 FOLLOWER]: Leader election won for term 6
I20250830 01:58:42.268282 29039 raft_consensus.cc:695] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 6 LEADER]: Becoming Leader. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Running, Role: LEADER
I20250830 01:58:42.268850 29039 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 5.26, Last appended by leader: 26, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:42.274487 29069 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 6 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:42.274736 29069 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:42.275381 29073 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:42.277858 29073 catalog_manager.cc:671] Loaded metadata for table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]
I20250830 01:58:42.278307 29073 catalog_manager.cc:671] Loaded metadata for table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]
I20250830 01:58:42.278621 29073 catalog_manager.cc:671] Loaded metadata for table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]
I20250830 01:58:42.278934 29073 catalog_manager.cc:671] Loaded metadata for table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]
I20250830 01:58:42.280710 29073 tablet_loader.cc:96] loaded metadata for tablet 18812772b9644792ac88671ff697e2a4 (table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a])
I20250830 01:58:42.281213 29073 tablet_loader.cc:96] loaded metadata for tablet 47a4f27ca16d436386ed86810020f84b (table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f])
I20250830 01:58:42.281670 29073 tablet_loader.cc:96] loaded metadata for tablet 7f45870df10d4910a749167310afb2e5 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368])
I20250830 01:58:42.281986 29073 tablet_loader.cc:96] loaded metadata for tablet 876295e43d7c4b01b2f772a492bc27cf (table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46])
I20250830 01:58:42.282617 29073 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:42.283555 29073 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:42.283627 29073 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:42.284864 29073 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:42.285720 29073 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Loaded TSK: 0
I20250830 01:58:42.286610 29073 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:42.287468 29073 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:42.288293 29073 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250830 01:58:42.314260 29000 master_service.cc:948] ReplaceTablet: received request to replace tablet 47a4f27ca16d436386ed86810020f84b from {username='slave'} at 127.0.0.1:56504
I20250830 01:58:42.314356 28999 master_service.cc:948] ReplaceTablet: received request to replace tablet 18812772b9644792ac88671ff697e2a4 from {username='slave'} at 127.0.0.1:56504
I20250830 01:58:42.315321 28995 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56504:
table { table_name: "default.Table_488d167c0efe4d1b91a27f563856f9f6" } new_table_name: "default.Table_59d8d3858b8c490f984614c9187eef7c" modify_external_catalogs: true
I20250830 01:58:42.315531 28994 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56504:
name: "default.Table_cef396d7e593458aa7679269acecf621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:42.316008 28996 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:56504
I20250830 01:58:42.316605 28991 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56504:
table { table_name: "default.Table_2c4fc026b80045d8a58497a870104278" } new_table_name: "default.Table_f8d128ad097b4e25a4cc03e5b2f02676" modify_external_catalogs: true
I20250830 01:58:42.316727 28993 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56504:
name: "default.Table_a36b1b6bfc854dfc907635b41d47be02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:42.316923 28996 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56504:
name: "default.Table_509af6139e2e4ba6bc5c537af315df6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:42.317288 28992 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56504:
table { table_name: "default.Table_2c93d305666e48d5b919f93ebe9e05ec" } new_table_name: "default.Table_942f583f6001419a91d24f8fdb74108c" modify_external_catalogs: true
I20250830 01:58:42.317453 28997 master_service.cc:948] ReplaceTablet: received request to replace tablet 18812772b9644792ac88671ff697e2a4 from {username='slave'} at 127.0.0.1:56504
I20250830 01:58:42.317775 28998 master_service.cc:948] ReplaceTablet: received request to replace tablet 7f45870df10d4910a749167310afb2e5 from {username='slave'} at 127.0.0.1:56504
I20250830 01:58:42.323271 28831 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Refusing update from remote peer ad9824f7e6604077ac8a034879b4c731: Log matching property violated. Preceding OpId in replica: term: 5 index: 26. Preceding OpId from leader: term: 6 index: 29. (index mismatch)
F20250830 01:58:42.324965 29052 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 5 received events: NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_488d167c0efe4d1b91a27f563856f9f6, message={"server":"","servicePrincipal":"","db":"default","table":"table_488d167c0efe4d1b91a27f563856f9f6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_488d167c0efe4d1b91a27f563856f9f6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_488d167c0efe4d1b91a27f563856f9f6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_488d167c0efe4d1b91a27f563856f9f6\",\"kudu.table_id\":\"a66ba6403ba349f3ade7628ab65cd30f\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_badff657ae07402a88632f78ec1befb7, message={"server":"","servicePrincipal":"","db":"default","table":"table_badff657ae07402a88632f78ec1befb7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_badff657ae07402a88632f78ec1befb7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_badff657ae07402a88632f78ec1befb7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_badff657ae07402a88632f78ec1befb7\",\"kudu.table_id\":\"7b416700773a4e27bf152c666f48a272\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250830 01:58:42.326431 29039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
@ 0x7fe837ad0dcd google::LogMessage::Fail() at ??:0
W20250830 01:58:42.329124 28986 consensus_peers.cc:489] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 -> Peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Couldn't send request to peer fd4f5c1e7015460e99715d8a2edd2959. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fe837ad4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe837ad07cc google::LogMessage::Flush() at ??:0
@ 0x7fe837ad0a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe84df05794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fe84df04041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250830 01:58:42.354269 29076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 6 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:42.354552 29076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fe84df0bcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fe84df0bb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fe83a82dcc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe83dc4f6db start_thread at ??:0
@ 0x7fe83698171f clone at ??:0
I20250830 01:58:42.375903 29076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:42.376142 29076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: FOLLOWER
W20250830 01:58:42.541692 27865 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:42.541774 28807 connection.cc:537] server connection from 127.0.0.1:55804 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:42.542090 27985 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:42.542655 27985 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.26.105.254:36635)
W20250830 01:58:42.542929 27985 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.105.254:36635)
W20250830 01:58:42.543251 27985 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.26.105.254:36635)
I20250830 01:58:42.857674 29076 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:42.857874 29076 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:42.858597 29076 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:42.860940 28808 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:42.861120 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:42.861274 28807 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:42.861547 29076 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250830 01:58:43.484872 29105 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:43.485098 29105 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:43.485756 29105 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:43.488432 28808 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:43.488606 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:43.488732 28807 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:43.488956 29105 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250830 01:58:43.495501 28790 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:58:43.495682 28790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:44.000121 29111 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:44.000331 29111 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:44.000988 29111 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:44.002875 28808 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:44.003058 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:44.003182 28807 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e; no voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:44.003403 29111 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250830 01:58:44.487617 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--webserver_interface=127.26.105.254
--webserver_port=37963
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:44.830504 29120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:44.830844 29120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:44.830966 29120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:44.839517 29120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:44.839628 29120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:44.839722 29120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:44.839800 29120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:44.850117 29120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.105.254
--webserver_port=37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:44.851842 29120 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:44.853696 29120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:44.862690 29129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:44.862720 29128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:44.864078 29131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:44.864886 29120 server_base.cc:1047] running on GCE node
I20250830 01:58:44.865657 29120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:44.867126 29120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:44.868331 29120 hybrid_clock.cc:648] HybridClock initialized: now 1756519124868293 us; error 28 us; skew 500 ppm
I20250830 01:58:44.868683 29120 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:44.871260 29120 webserver.cc:480] Webserver started at http://127.26.105.254:37963/ using document root <none> and password file <none>
I20250830 01:58:44.872172 29120 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:44.872273 29120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:44.877859 29120 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20250830 01:58:44.880934 29137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:44.882368 29120 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250830 01:58:44.882534 29120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
uuid: "ad9824f7e6604077ac8a034879b4c731"
format_stamp: "Formatted at 2025-08-30 01:58:19 on dist-test-slave-gp2r"
I20250830 01:58:44.883260 29120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:44.919070 29120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:44.920012 29120 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:44.920311 29120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:44.941356 29120 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.254:36635
I20250830 01:58:44.941377 29189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.254:36635 every 8 connection(s)
I20250830 01:58:44.943121 29120 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-0/data/info.pb
I20250830 01:58:44.947835 29191 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:44.948105 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 29120
I20250830 01:58:44.948074 29191 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:44.955013 29191 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:44.956853 29190 sys_catalog.cc:263] Verifying existing consensus state
W20250830 01:58:44.961979 28807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20250830 01:58:44.964310 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:44.964355 29190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap starting.
I20250830 01:58:44.977715 29190 log.cc:826] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:45.002766 29190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=28 ignored=0} inserts{seen=17 ignored=0} mutations{seen=15 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250830 01:58:45.003618 29190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Bootstrap complete.
I20250830 01:58:45.009379 29190 raft_consensus.cc:357] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 6 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:45.012156 29190 raft_consensus.cc:738] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Initialized, Role: FOLLOWER
I20250830 01:58:45.012827 29190 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 6.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:45.013612 29200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:45.013940 29200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:45.014779 29190 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:45.014834 29165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:45.015394 29165 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba697035f304124a88a1cf1aabb748e in term 6.
I20250830 01:58:45.016116 28808 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters: fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:45.016462 29191 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250830 01:58:45.016566 29191 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Starting leader election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:45.016634 29191 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 6 FOLLOWER]: Advancing to term 7
I20250830 01:58:45.018615 29191 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:45.019141 29191 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 election: Requested vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:45.019790 29165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ba697035f304124a88a1cf1aabb748e" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731"
I20250830 01:58:45.020048 29165 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 6 FOLLOWER]: Advancing to term 7
W20250830 01:58:45.020551 28807 leader_election.cc:336] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:45.025640 29165 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba697035f304124a88a1cf1aabb748e in term 7.
I20250830 01:58:45.026120 28808 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731; no voters: fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:45.026501 29191 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 7 FOLLOWER]: Leader election won for term 7
I20250830 01:58:45.026978 29191 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [term 7 LEADER]: Becoming Leader. State: Replica: 9ba697035f304124a88a1cf1aabb748e, State: Running, Role: LEADER
I20250830 01:58:45.027417 29191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 6.30, Last appended by leader: 30, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:45.030041 29211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ba697035f304124a88a1cf1aabb748e. Latest consensus state: current_term: 7 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:45.030373 29211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:45.062310 27700 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:45.065272 29155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:57775; Asking this server to re-register.
I20250830 01:58:45.066483 27700 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:45.068418 29155 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:45.085448 29215 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:45.085731 29215 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: loading cluster ID for follower catalog manager: success
I20250830 01:58:45.082465 29120 master_runner.cc:186] Error getting master registration for 127.26.105.252:46233: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111),
I20250830 01:58:45.087746 29120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:45.089593 29215 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: acquiring CA information for follower catalog manager: success
I20250830 01:58:45.091738 29215 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:45.155275 27839 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:45.156796 29155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:51435; Asking this server to re-register.
I20250830 01:58:45.157415 27839 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:45.157437 27975 heartbeater.cc:344] Connected to a master server at 127.26.105.254:36635
I20250830 01:58:45.158910 29154 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:45.159838 29155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:60103; Asking this server to re-register.
I20250830 01:58:45.160441 27975 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:45.161837 29155 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:45.543069 29165 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Refusing update from remote peer 9ba697035f304124a88a1cf1aabb748e: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 7 index: 31. (index mismatch)
I20250830 01:58:45.544175 29211 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 28, Time since last communication: 0.000s
I20250830 01:58:45.547449 29227 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ba697035f304124a88a1cf1aabb748e. Latest consensus state: current_term: 7 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:45.547708 29227 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:45.550278 29211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:45.550362 29227 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "9ba697035f304124a88a1cf1aabb748e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:45.550511 29211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:45.550593 29227 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: FOLLOWER
W20250830 01:58:45.553031 28807 consensus_peers.cc:489] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e -> Peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Couldn't send request to peer fd4f5c1e7015460e99715d8a2edd2959. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250830 01:58:45.593622 29218 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:45.596216 29218 catalog_manager.cc:671] Loaded metadata for table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]
I20250830 01:58:45.596648 29218 catalog_manager.cc:671] Loaded metadata for table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]
I20250830 01:58:45.596966 29218 catalog_manager.cc:671] Loaded metadata for table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]
I20250830 01:58:45.597263 29218 catalog_manager.cc:671] Loaded metadata for table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]
I20250830 01:58:45.599258 29218 tablet_loader.cc:96] loaded metadata for tablet 2b91ee2988c84569b779ea9292dcaad6 (table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f])
I20250830 01:58:45.599509 29218 tablet_loader.cc:96] loaded metadata for tablet 320d5ead70554ba6a39212ea95a6ffb3 (table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a])
I20250830 01:58:45.600394 29218 tablet_loader.cc:96] loaded metadata for tablet 876295e43d7c4b01b2f772a492bc27cf (table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46])
I20250830 01:58:45.600788 29218 tablet_loader.cc:96] loaded metadata for tablet dcd2a38e964f4d5fab198259e0686378 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368])
I20250830 01:58:45.601176 29218 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:45.602100 29218 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:45.602201 29218 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:45.603356 29218 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:45.604182 29218 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9ba697035f304124a88a1cf1aabb748e: Loaded TSK: 0
I20250830 01:58:45.604990 29218 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:45.605809 29218 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:45.606599 29218 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250830 01:58:45.612421 28820 master_service.cc:948] ReplaceTablet: received request to replace tablet 3996bdda030340d081af363dbdfea328 from {username='slave'} at 127.0.0.1:55772
I20250830 01:58:45.612866 28821 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55772:
name: "default.Table_624ff0828e534d54a8ea5f69551f7e8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:45.612902 28819 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55772:
name: "default.Table_739ceee58b144d72b56342855499ecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:45.613930 28817 master_service.cc:948] ReplaceTablet: received request to replace tablet 18812772b9644792ac88671ff697e2a4 from {username='slave'} at 127.0.0.1:55772
I20250830 01:58:45.614082 28820 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55772:
table { table_name: "default.Table_488d167c0efe4d1b91a27f563856f9f6" } new_table_name: "default.Table_59d8d3858b8c490f984614c9187eef7c" modify_external_catalogs: true
I20250830 01:58:45.614132 28818 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55772:
table { table_name: "default.Table_d19bb3dab12e478cbd8897446f84d508" } new_table_name: "default.Table_31e5e45019fd41888d6a471fbd3736b4" modify_external_catalogs: true
I20250830 01:58:45.614713 28817 master_service.cc:948] ReplaceTablet: received request to replace tablet 7f45870df10d4910a749167310afb2e5 from {username='slave'} at 127.0.0.1:55772
I20250830 01:58:45.614908 28816 master_service.cc:948] ReplaceTablet: received request to replace tablet 47a4f27ca16d436386ed86810020f84b from {username='slave'} at 127.0.0.1:55772
I20250830 01:58:45.614987 28815 master_service.cc:948] ReplaceTablet: received request to replace tablet 18812772b9644792ac88671ff697e2a4 from {username='slave'} at 127.0.0.1:55772
I20250830 01:58:45.615887 28816 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55772:
name: "default.Table_a36b1b6bfc854dfc907635b41d47be02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:45.615795 28815 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55772:
name: "default.Table_cef396d7e593458aa7679269acecf621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:45.616370 28813 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55772:
name: "default.Table_509af6139e2e4ba6bc5c537af315df6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:45.616573 28817 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55772:
table { table_name: "default.Table_2c93d305666e48d5b919f93ebe9e05ec" } new_table_name: "default.Table_942f583f6001419a91d24f8fdb74108c" modify_external_catalogs: true
I20250830 01:58:45.616739 28814 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55772:
table { table_name: "default.Table_2c4fc026b80045d8a58497a870104278" } new_table_name: "default.Table_f8d128ad097b4e25a4cc03e5b2f02676" modify_external_catalogs: true
I20250830 01:58:45.618105 28812 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55772:
name: "default.Table_ac070120b97141e188efab1e24ece67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250830 01:58:45.619771 28881 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 5 received events: NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_488d167c0efe4d1b91a27f563856f9f6, message={"server":"","servicePrincipal":"","db":"default","table":"table_488d167c0efe4d1b91a27f563856f9f6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_488d167c0efe4d1b91a27f563856f9f6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_488d167c0efe4d1b91a27f563856f9f6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_488d167c0efe4d1b91a27f563856f9f6\",\"kudu.table_id\":\"a66ba6403ba349f3ade7628ab65cd30f\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_badff657ae07402a88632f78ec1befb7, message={"server":"","servicePrincipal":"","db":"default","table":"table_badff657ae07402a88632f78ec1befb7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_badff657ae07402a88632f78ec1befb7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_badff657ae07402a88632f78ec1befb7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_badff657ae07402a88632f78ec1befb7\",\"kudu.table_id\":\"7b416700773a4e27bf152c666f48a272\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f265eabedcd google::LogMessage::Fail() at ??:0
@ 0x7f265eac2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f265eabe7cc google::LogMessage::Flush() at ??:0
@ 0x7f265eabea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f2674ef3794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f2674ef2041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f2674ef9cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f2674ef9b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f266181bcc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2664c3d6db start_thread at ??:0
@ 0x7f265d96f71f clone at ??:0
W20250830 01:58:45.837278 27984 connection.cc:537] client connection to 127.26.105.253:39521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:45.838014 27984 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.26.105.253:39521)
W20250830 01:58:45.838613 27984 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.105.253:39521)
W20250830 01:58:45.838930 27984 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.26.105.253:39521)
I20250830 01:58:46.049780 29227 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:46.050029 29227 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:46.051874 29227 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:46.053182 29140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:46.054414 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:46.054667 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:46.054767 29140 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:46.055049 29227 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250830 01:58:46.661027 29261 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:46.661245 29261 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:46.661883 29261 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:46.664160 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:46.664355 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:46.664474 29140 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:46.664796 29261 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250830 01:58:47.262219 29271 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:47.262408 29271 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:47.263024 29271 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:47.265123 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:47.265470 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:47.265579 29140 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:47.265807 29271 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250830 01:58:47.813491 29279 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:47.813669 29279 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:47.814417 29279 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:47.816390 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:47.816560 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233): Network error: Client connection negotiation failed: client connection to 127.26.105.252:46233: connect: Connection refused (error 111)
I20250830 01:58:47.816648 29140 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731; no voters: 9ba697035f304124a88a1cf1aabb748e, fd4f5c1e7015460e99715d8a2edd2959
I20250830 01:58:47.816864 29279 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250830 01:58:47.939548 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--webserver_interface=127.26.105.252
--webserver_port=39345
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:48.067032 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 11: this message will repeat every 5th retry.
W20250830 01:58:48.275091 29283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:48.275440 29283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:48.275564 29283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:48.285733 29283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:48.285841 29283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:48.285941 29283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:48.286005 29283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:48.296185 29283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.105.252
--webserver_port=39345
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:48.297852 29283 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:48.299688 29283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:48.310075 29294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:48.310022 29295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:48.310379 29297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:48.311280 29283 server_base.cc:1047] running on GCE node
I20250830 01:58:48.312036 29283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:48.313413 29283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:48.314647 29283 hybrid_clock.cc:648] HybridClock initialized: now 1756519128314589 us; error 45 us; skew 500 ppm
I20250830 01:58:48.315084 29283 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:48.317497 29283 webserver.cc:480] Webserver started at http://127.26.105.252:39345/ using document root <none> and password file <none>
I20250830 01:58:48.318313 29283 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:48.318416 29283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:48.323822 29283 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250830 01:58:48.326817 29303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:48.328217 29283 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250830 01:58:48.328372 29283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
uuid: "fd4f5c1e7015460e99715d8a2edd2959"
format_stamp: "Formatted at 2025-08-30 01:58:20 on dist-test-slave-gp2r"
I20250830 01:58:48.329057 29283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:48.377724 29283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:48.378612 29283 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:48.378896 29283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:48.398444 29283 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.252:46233
I20250830 01:58:48.398476 29354 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.252:46233 every 8 connection(s)
I20250830 01:58:48.400106 29283 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
I20250830 01:58:48.406500 29355 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:48.409862 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap starting.
I20250830 01:58:48.409930 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 29283
I20250830 01:58:48.420585 29355 log.cc:826] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:48.442524 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=23 ignored=0} inserts{seen=15 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250830 01:58:48.443060 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap complete.
I20250830 01:58:48.448226 29355 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:48.448915 29355 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4f5c1e7015460e99715d8a2edd2959, State: Initialized, Role: FOLLOWER
I20250830 01:58:48.449734 29355 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 4.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:48.450590 29359 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:48.450819 29359 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:48.451596 29355 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:48.468122 29373 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:48.468636 29373 catalog_manager.cc:1554] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: loading cluster ID for follower catalog manager: success
I20250830 01:58:48.476780 29373 catalog_manager.cc:1576] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: acquiring CA information for follower catalog manager: success
I20250830 01:58:48.480322 29373 catalog_manager.cc:1604] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:48.487429 29283 master_runner.cc:186] Error getting master registration for 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111),
I20250830 01:58:48.491004 29283 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:48.556172 27699 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:48.559000 29320 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:39717; Asking this server to re-register.
I20250830 01:58:48.559664 27699 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:48.561151 29320 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:48.651798 27838 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:48.652041 27838 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
I20250830 01:58:48.654143 29320 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:46951; Asking this server to re-register.
I20250830 01:58:48.654835 27838 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:48.656235 29320 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:48.814391 27974 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:48.814600 27974 heartbeater.cc:499] Master 127.26.105.252:46233 was elected leader, sending a full tablet report...
I20250830 01:58:48.816184 29320 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:43067; Asking this server to re-register.
I20250830 01:58:48.816727 27974 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:48.817906 29320 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:48.849026 29359 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:48.849291 29359 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:48.851373 29359 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
I20250830 01:58:48.851783 29165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:48.852046 29165 raft_consensus.cc:2366] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd4f5c1e7015460e99715d8a2edd2959 for earlier term 5. Current term is 7.
W20250830 01:58:48.851991 29306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:48.852834 29307 leader_election.cc:400] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd4f5c1e7015460e99715d8a2edd2959 for earlier term 5. Current term is 7.
I20250830 01:58:48.852942 29307 leader_election.cc:403] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250830 01:58:48.853224 29359 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 4 FOLLOWER]: Advancing to term 7
W20250830 01:58:48.853431 29306 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:48.856897 29359 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 7 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd4f5c1e7015460e99715d8a2edd2959 for earlier term 5. Current term is 7.
W20250830 01:58:49.318535 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 11: this message will repeat every 5th retry.
I20250830 01:58:49.590014 29390 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:49.590245 29390 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:49.590902 29390 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
I20250830 01:58:49.591331 29165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4f5c1e7015460e99715d8a2edd2959" candidate_term: 8 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "ad9824f7e6604077ac8a034879b4c731" is_pre_election: true
I20250830 01:58:49.591567 29165 raft_consensus.cc:2408] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd4f5c1e7015460e99715d8a2edd2959 for term 8 because replica has last-logged OpId of term: 7 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 23.
W20250830 01:58:49.592772 29306 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:49.592877 29306 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:49.593147 29390 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250830 01:58:50.056147 29400 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:50.056357 29400 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:50.056998 29400 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
W20250830 01:58:50.059459 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:50.063773 29330 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 8 candidate_status { last_received { term: 7 index: 31 } } ignore_live_leader: false dest_uuid: "fd4f5c1e7015460e99715d8a2edd2959" is_pre_election: true
I20250830 01:58:50.064180 29330 raft_consensus.cc:2466] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 7.
I20250830 01:58:50.064635 29140 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e
I20250830 01:58:50.064905 29400 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250830 01:58:50.065012 29400 raft_consensus.cc:491] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9ba697035f304124a88a1cf1aabb748e)
I20250830 01:58:50.065114 29400 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 7 FOLLOWER]: Advancing to term 8
I20250830 01:58:50.067327 29400 raft_consensus.cc:513] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:50.067929 29400 leader_election.cc:290] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 election: Requested vote from peers 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521), fd4f5c1e7015460e99715d8a2edd2959 (127.26.105.252:46233)
I20250830 01:58:50.068615 29330 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad9824f7e6604077ac8a034879b4c731" candidate_term: 8 candidate_status { last_received { term: 7 index: 31 } } ignore_live_leader: false dest_uuid: "fd4f5c1e7015460e99715d8a2edd2959"
I20250830 01:58:50.068775 29330 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 7 FOLLOWER]: Advancing to term 8
W20250830 01:58:50.069233 29140 leader_election.cc:336] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:50.070938 29330 raft_consensus.cc:2466] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9824f7e6604077ac8a034879b4c731 in term 8.
I20250830 01:58:50.071360 29140 leader_election.cc:304] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad9824f7e6604077ac8a034879b4c731, fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e
I20250830 01:58:50.071650 29400 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 8 FOLLOWER]: Leader election won for term 8
I20250830 01:58:50.072103 29400 raft_consensus.cc:695] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [term 8 LEADER]: Becoming Leader. State: Replica: ad9824f7e6604077ac8a034879b4c731, State: Running, Role: LEADER
I20250830 01:58:50.072662 29400 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 7.31, Last appended by leader: 31, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:50.074781 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 8 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:50.075358 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:50.076067 29407 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250830 01:58:50.078631 29407 catalog_manager.cc:671] Loaded metadata for table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]
I20250830 01:58:50.079095 29407 catalog_manager.cc:671] Loaded metadata for table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46]
I20250830 01:58:50.079448 29407 catalog_manager.cc:671] Loaded metadata for table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]
I20250830 01:58:50.079787 29407 catalog_manager.cc:671] Loaded metadata for table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]
I20250830 01:58:50.081817 29407 tablet_loader.cc:96] loaded metadata for tablet 2b91ee2988c84569b779ea9292dcaad6 (table default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f])
I20250830 01:58:50.082115 29407 tablet_loader.cc:96] loaded metadata for tablet 320d5ead70554ba6a39212ea95a6ffb3 (table default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a])
I20250830 01:58:50.083051 29407 tablet_loader.cc:96] loaded metadata for tablet 876295e43d7c4b01b2f772a492bc27cf (table default.table_2c4fc026b80045d8a58497a870104278 [id=65cc6cabcfd4405994a16c620558df46])
I20250830 01:58:50.083492 29407 tablet_loader.cc:96] loaded metadata for tablet dcd2a38e964f4d5fab198259e0686378 (table default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368])
I20250830 01:58:50.083899 29407 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250830 01:58:50.084826 29407 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:50.084897 29407 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250830 01:58:50.086091 29407 catalog_manager.cc:1506] Loading token signing keys...
I20250830 01:58:50.086970 29407 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731: Loaded TSK: 0
I20250830 01:58:50.088043 29407 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250830 01:58:50.089305 29407 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250830 01:58:50.090080 29120 master_runner.cc:186] Error getting master registration for 127.26.105.253:39521: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111),
I20250830 01:58:50.090292 29120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:50.090478 29407 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
W20250830 01:58:50.100073 29140 consensus_peers.cc:489] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 -> Peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Couldn't send request to peer 9ba697035f304124a88a1cf1aabb748e. Status: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250830 01:58:50.100114 29330 raft_consensus.cc:1273] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Refusing update from remote peer ad9824f7e6604077ac8a034879b4c731: Log matching property violated. Preceding OpId in replica: term: 4 index: 23. Preceding OpId from leader: term: 8 index: 33. (index mismatch)
I20250830 01:58:50.100808 29400 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 23, Time since last communication: 0.000s
I20250830 01:58:50.109200 29414 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:50.109575 29414 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:50.109844 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:50.109922 29412 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:50.110283 29412 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:50.110143 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:50.110466 29409 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9824f7e6604077ac8a034879b4c731. Latest consensus state: current_term: 8 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:50.110659 29409 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:50.111506 29408 mvcc.cc:204] Tried to move back new op lower bound from 7194702356880863232 to 7194702356781760512. Current Snapshot: MvccSnapshot[applied={T|T < 7194702356880863232}]
I20250830 01:58:50.112915 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:50.113711 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9824f7e6604077ac8a034879b4c731 [sys.catalog]: This master's current role is: LEADER
I20250830 01:58:50.114291 29409 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad9824f7e6604077ac8a034879b4c731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:50.114470 29409 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:50.116871 29410 mvcc.cc:204] Tried to move back new op lower bound from 7194702356880863232 to 7194702303579705344. Current Snapshot: MvccSnapshot[applied={T|T < 7194702356880863232}]
I20250830 01:58:50.120751 29152 master_service.cc:948] ReplaceTablet: received request to replace tablet 47a4f27ca16d436386ed86810020f84b from {username='slave'} at 127.0.0.1:57218
I20250830 01:58:50.121500 29155 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:57218
I20250830 01:58:50.121558 29149 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57218:
name: "default.Table_badff657ae07402a88632f78ec1befb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:50.122056 29153 master_service.cc:948] ReplaceTablet: received request to replace tablet fc86731b9fa247619aa9414e3709ba35 from {username='slave'} at 127.0.0.1:57218
I20250830 01:58:50.121950 29152 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57218:
name: "default.Table_739ceee58b144d72b56342855499ecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:50.122685 29151 master_service.cc:948] ReplaceTablet: received request to replace tablet 177a8293c20c412ea767995c5c9af204 from {username='slave'} at 127.0.0.1:57218
I20250830 01:58:50.122649 29150 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57218:
name: "default.Table_1ad78e45bc5842aeaf69a74b3ef38007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:50.123415 29148 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57218:
name: "default.Table_11641b1a96984fa099ffa232b6a0033e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:50.124197 29147 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57218:
table { table_name: "default.Table_488d167c0efe4d1b91a27f563856f9f6" } new_table_name: "default.Table_59d8d3858b8c490f984614c9187eef7c" modify_external_catalogs: true
I20250830 01:58:50.124702 29146 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57218:
table { table_name: "default.Table_d19bb3dab12e478cbd8897446f84d508" } new_table_name: "default.Table_31e5e45019fd41888d6a471fbd3736b4" modify_external_catalogs: true
I20250830 01:58:50.125172 29154 master_service.cc:948] ReplaceTablet: received request to replace tablet 47a4f27ca16d436386ed86810020f84b from {username='slave'} at 127.0.0.1:57218
I20250830 01:58:50.126434 29151 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57218:
name: "default.Table_cef396d7e593458aa7679269acecf621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:50.127060 29153 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57218:
name: "default.Table_a36b1b6bfc854dfc907635b41d47be02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:50.127748 29154 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57218:
name: "default.Table_624ff0828e534d54a8ea5f69551f7e8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250830 01:58:50.128520 29155 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57218:
table { table_name: "default.Table_2c93d305666e48d5b919f93ebe9e05ec" } new_table_name: "default.Table_942f583f6001419a91d24f8fdb74108c" modify_external_catalogs: true
F20250830 01:58:50.141006 29214 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 5 received events: NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_488d167c0efe4d1b91a27f563856f9f6, message={"server":"","servicePrincipal":"","db":"default","table":"table_488d167c0efe4d1b91a27f563856f9f6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_488d167c0efe4d1b91a27f563856f9f6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_488d167c0efe4d1b91a27f563856f9f6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_488d167c0efe4d1b91a27f563856f9f6\",\"kudu.table_id\":\"a66ba6403ba349f3ade7628ab65cd30f\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756519102, eventType=CREATE_TABLE, dbName=default, tableName=table_badff657ae07402a88632f78ec1befb7, message={"server":"","servicePrincipal":"","db":"default","table":"table_badff657ae07402a88632f78ec1befb7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_badff657ae07402a88632f78ec1befb7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756519102},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/warehouse/table_badff657ae07402a88632f78ec1befb7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_badff657ae07402a88632f78ec1befb7\",\"kudu.table_id\":\"7b416700773a4e27bf152c666f48a272\",\"transient_lastDdlTime\":\"1756519102\",\"comment\":\"\",\"kudu.master_addresses\":\"127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"b3f54d3a81d9420093a5097ef44cfe8f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756519102,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250830 01:58:50.144133 27772 tablet_service.cc:1468] Processing CreateTablet for tablet 2b91ee2988c84569b779ea9292dcaad6 (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.144169 27773 tablet_service.cc:1468] Processing CreateTablet for tablet 320d5ead70554ba6a39212ea95a6ffb3 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.144554 27772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b91ee2988c84569b779ea9292dcaad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.144789 27773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320d5ead70554ba6a39212ea95a6ffb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.145125 27771 tablet_service.cc:1468] Processing CreateTablet for tablet dcd2a38e964f4d5fab198259e0686378 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.145808 27771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcd2a38e964f4d5fab198259e0686378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.155436 29423 tablet_bootstrap.cc:492] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:50.157655 27908 tablet_service.cc:1468] Processing CreateTablet for tablet 320d5ead70554ba6a39212ea95a6ffb3 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.157716 27906 tablet_service.cc:1468] Processing CreateTablet for tablet 2b91ee2988c84569b779ea9292dcaad6 (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.158061 27908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320d5ead70554ba6a39212ea95a6ffb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.158265 27906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b91ee2988c84569b779ea9292dcaad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.158677 29423 tablet_bootstrap.cc:654] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.158665 27907 tablet_service.cc:1468] Processing CreateTablet for tablet dcd2a38e964f4d5fab198259e0686378 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.159044 27907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcd2a38e964f4d5fab198259e0686378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.168906 27632 tablet_service.cc:1468] Processing CreateTablet for tablet 2b91ee2988c84569b779ea9292dcaad6 (DEFAULT_TABLE table=default.table_488d167c0efe4d1b91a27f563856f9f6 [id=a66ba6403ba349f3ade7628ab65cd30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.169234 27633 tablet_service.cc:1468] Processing CreateTablet for tablet dcd2a38e964f4d5fab198259e0686378 (DEFAULT_TABLE table=default.table_2c93d305666e48d5b919f93ebe9e05ec [id=32148c9d0fcd4e8eb909814b954bc368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.169504 29425 tablet_bootstrap.cc:492] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:50.169580 27632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b91ee2988c84569b779ea9292dcaad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.168951 27631 tablet_service.cc:1468] Processing CreateTablet for tablet 320d5ead70554ba6a39212ea95a6ffb3 (DEFAULT_TABLE table=default.table_d19bb3dab12e478cbd8897446f84d508 [id=e9fceb6dca37466384f24c8bde25f71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250830 01:58:50.169884 27631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320d5ead70554ba6a39212ea95a6ffb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.170310 27633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcd2a38e964f4d5fab198259e0686378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250830 01:58:50.172149 29425 tablet_bootstrap.cc:654] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.174888 29423 tablet_bootstrap.cc:492] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:50.175136 29423 ts_tablet_manager.cc:1397] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.012s
I20250830 01:58:50.176138 29423 raft_consensus.cc:357] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.176497 29423 raft_consensus.cc:383] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.176633 29423 raft_consensus.cc:738] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.176957 29423 consensus_queue.cc:260] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.181325 29423 ts_tablet_manager.cc:1428] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250830 01:58:50.181840 29423 tablet_bootstrap.cc:492] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
@ 0x7feec0ee0dcd google::LogMessage::Fail() at ??:0
@ 0x7feec0ee4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7feec0ee07cc google::LogMessage::Flush() at ??:0
@ 0x7feec0ee0a11 google::LogMessage::~LogMessage() at ??:0
I20250830 01:58:50.193703 29427 tablet_bootstrap.cc:492] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:50.194664 29425 tablet_bootstrap.cc:492] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:50.194824 29425 ts_tablet_manager.cc:1397] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.015s
I20250830 01:58:50.195708 29425 raft_consensus.cc:357] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.195900 29425 raft_consensus.cc:383] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.196013 29425 raft_consensus.cc:738] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
@ 0x7feed7315794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250830 01:58:50.196244 29425 consensus_queue.cc:260] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.196417 29427 tablet_bootstrap.cc:654] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.196801 29423 tablet_bootstrap.cc:654] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.196835 29425 ts_tablet_manager.cc:1428] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250830 01:58:50.197415 29425 tablet_bootstrap.cc:492] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:50.199337 29425 tablet_bootstrap.cc:654] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.202069 29423 tablet_bootstrap.cc:492] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:50.202275 29423 ts_tablet_manager.cc:1397] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.002s
I20250830 01:58:50.203609 29423 raft_consensus.cc:357] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.203699 29427 tablet_bootstrap.cc:492] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:50.203809 29423 raft_consensus.cc:383] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.203877 29427 ts_tablet_manager.cc:1397] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20250830 01:58:50.203892 29423 raft_consensus.cc:738] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.204149 29423 consensus_queue.cc:260] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.204636 29423 ts_tablet_manager.cc:1428] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250830 01:58:50.204943 29423 tablet_bootstrap.cc:492] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca: Bootstrap starting.
I20250830 01:58:50.206617 29423 tablet_bootstrap.cc:654] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.212548 29427 raft_consensus.cc:357] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.212846 29427 raft_consensus.cc:383] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.212950 29427 raft_consensus.cc:738] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.213294 29427 consensus_queue.cc:260] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.218050 29427 ts_tablet_manager.cc:1428] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s
I20250830 01:58:50.218528 29427 tablet_bootstrap.cc:492] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:50.220954 29427 tablet_bootstrap.cc:654] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.223388 29427 tablet_bootstrap.cc:492] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:50.223378 29425 tablet_bootstrap.cc:492] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:50.223541 29427 ts_tablet_manager.cc:1397] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250830 01:58:50.223541 29425 ts_tablet_manager.cc:1397] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.025s
I20250830 01:58:50.224545 29427 raft_consensus.cc:357] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.224751 29427 raft_consensus.cc:383] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.224723 29425 raft_consensus.cc:357] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.224828 29427 raft_consensus.cc:738] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.224915 29425 raft_consensus.cc:383] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.224999 29425 raft_consensus.cc:738] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.225188 29427 consensus_queue.cc:260] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.225261 29425 consensus_queue.cc:260] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.225760 29425 ts_tablet_manager.cc:1428] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250830 01:58:50.226037 29425 tablet_bootstrap.cc:492] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822: Bootstrap starting.
I20250830 01:58:50.227804 29425 tablet_bootstrap.cc:654] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822: Neither blocks nor log segments found. Creating new log.
@ 0x7feed7314041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250830 01:58:50.237008 29427 ts_tablet_manager.cc:1428] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250830 01:58:50.241354 29427 tablet_bootstrap.cc:492] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227: Bootstrap starting.
I20250830 01:58:50.243793 29427 tablet_bootstrap.cc:654] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227: Neither blocks nor log segments found. Creating new log.
I20250830 01:58:50.247589 29425 tablet_bootstrap.cc:492] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822: No bootstrap required, opened a new log
I20250830 01:58:50.247586 29427 tablet_bootstrap.cc:492] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227: No bootstrap required, opened a new log
I20250830 01:58:50.247743 29425 ts_tablet_manager.cc:1397] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.013s
I20250830 01:58:50.247740 29427 ts_tablet_manager.cc:1397] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250830 01:58:50.248543 29425 raft_consensus.cc:357] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.248538 29427 raft_consensus.cc:357] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.248726 29427 raft_consensus.cc:383] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.248732 29425 raft_consensus.cc:383] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.248817 29427 raft_consensus.cc:738] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d61552c37a44ad9a7a6f6f5c236227, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.248826 29425 raft_consensus.cc:738] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b0b641a51432f9172ed0b41521822, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.249110 29425 consensus_queue.cc:260] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.249094 29427 consensus_queue.cc:260] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.251400 29423 tablet_bootstrap.cc:492] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca: No bootstrap required, opened a new log
I20250830 01:58:50.251569 29423 ts_tablet_manager.cc:1397] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca: Time spent bootstrapping tablet: real 0.047s user 0.035s sys 0.010s
I20250830 01:58:50.252589 29423 raft_consensus.cc:357] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.252794 29423 raft_consensus.cc:383] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250830 01:58:50.252892 29425 ts_tablet_manager.cc:1428] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250830 01:58:50.252933 29427 ts_tablet_manager.cc:1428] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250830 01:58:50.252882 29423 raft_consensus.cc:738] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Initialized, Role: FOLLOWER
I20250830 01:58:50.253230 29423 consensus_queue.cc:260] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
@ 0x7feed731bcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250830 01:58:50.253793 29423 ts_tablet_manager.cc:1428] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
@ 0x7feed731bb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7feec3c3dcc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feec705f6db start_thread at ??:0
@ 0x7feebfd9171f clone at ??:0
I20250830 01:58:50.415753 29253 raft_consensus.cc:491] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:50.415961 29253 raft_consensus.cc:513] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.416898 29253 leader_election.cc:290] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:50.417384 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320d5ead70554ba6a39212ea95a6ffb3" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:50.417642 27652 raft_consensus.cc:2466] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:50.417793 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320d5ead70554ba6a39212ea95a6ffb3" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:50.417987 27929 raft_consensus.cc:2466] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:50.418416 27727 leader_election.cc:304] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:50.418715 29253 raft_consensus.cc:2802] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:50.418833 29253 raft_consensus.cc:491] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:50.418928 29253 raft_consensus.cc:3058] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.420795 29253 raft_consensus.cc:513] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.421353 29253 leader_election.cc:290] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:50.421900 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320d5ead70554ba6a39212ea95a6ffb3" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:50.421973 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320d5ead70554ba6a39212ea95a6ffb3" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:50.422119 27929 raft_consensus.cc:3058] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.422192 27652 raft_consensus.cc:3058] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.424953 27652 raft_consensus.cc:2466] T 320d5ead70554ba6a39212ea95a6ffb3 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:50.425388 27929 raft_consensus.cc:2466] T 320d5ead70554ba6a39212ea95a6ffb3 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:50.425446 27727 leader_election.cc:304] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:50.425707 29253 raft_consensus.cc:2802] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:50.426201 29253 raft_consensus.cc:695] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:50.426599 29253 consensus_queue.cc:237] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
W20250830 01:58:50.444408 27985 connection.cc:537] client connection to 127.26.105.254:36635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250830 01:58:50.445194 27985 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.26.105.254:36635)
W20250830 01:58:50.445837 27985 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.105.254:36635)
W20250830 01:58:50.446494 27985 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.26.105.254:36635)
I20250830 01:58:50.545467 29253 raft_consensus.cc:491] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:50.545676 29253 raft_consensus.cc:513] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.546324 29253 leader_election.cc:290] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:50.546866 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b91ee2988c84569b779ea9292dcaad6" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:50.546901 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b91ee2988c84569b779ea9292dcaad6" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:50.547153 27652 raft_consensus.cc:2466] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:50.547156 27929 raft_consensus.cc:2466] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:50.547590 27729 leader_election.cc:304] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5563a27e4e4748813d624aab13cdca, 893b0b641a51432f9172ed0b41521822; no voters:
I20250830 01:58:50.547863 29253 raft_consensus.cc:2802] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:50.547994 29253 raft_consensus.cc:491] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:50.548100 29253 raft_consensus.cc:3058] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.550010 29253 raft_consensus.cc:513] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.550709 29253 leader_election.cc:290] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:50.551159 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b91ee2988c84569b779ea9292dcaad6" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:50.551366 27652 raft_consensus.cc:3058] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.551401 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b91ee2988c84569b779ea9292dcaad6" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:50.551648 27929 raft_consensus.cc:3058] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.554250 27652 raft_consensus.cc:2466] T 2b91ee2988c84569b779ea9292dcaad6 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:50.554481 27929 raft_consensus.cc:2466] T 2b91ee2988c84569b779ea9292dcaad6 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:50.554795 27727 leader_election.cc:304] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:50.555106 29253 raft_consensus.cc:2802] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:50.555271 29253 raft_consensus.cc:695] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:50.555707 29253 consensus_queue.cc:237] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.586907 29253 raft_consensus.cc:491] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:50.587111 29253 raft_consensus.cc:513] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.587713 29253 leader_election.cc:290] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:50.588349 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcd2a38e964f4d5fab198259e0686378" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822" is_pre_election: true
I20250830 01:58:50.588321 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcd2a38e964f4d5fab198259e0686378" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227" is_pre_election: true
I20250830 01:58:50.588552 27929 raft_consensus.cc:2466] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:50.588586 27652 raft_consensus.cc:2466] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 0.
I20250830 01:58:50.588992 27727 leader_election.cc:304] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:50.589272 29253 raft_consensus.cc:2802] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250830 01:58:50.589401 29253 raft_consensus.cc:491] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250830 01:58:50.589505 29253 raft_consensus.cc:3058] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.591768 29253 raft_consensus.cc:513] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.592360 29253 leader_election.cc:290] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Requested vote from peers 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219), 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:50.592780 27652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcd2a38e964f4d5fab198259e0686378" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d61552c37a44ad9a7a6f6f5c236227"
I20250830 01:58:50.592816 27929 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcd2a38e964f4d5fab198259e0686378" candidate_uuid: "7f5563a27e4e4748813d624aab13cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893b0b641a51432f9172ed0b41521822"
I20250830 01:58:50.592962 27652 raft_consensus.cc:3058] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.593015 27929 raft_consensus.cc:3058] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822 [term 0 FOLLOWER]: Advancing to term 1
I20250830 01:58:50.594978 27652 raft_consensus.cc:2466] T dcd2a38e964f4d5fab198259e0686378 P 03d61552c37a44ad9a7a6f6f5c236227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:50.595028 27929 raft_consensus.cc:2466] T dcd2a38e964f4d5fab198259e0686378 P 893b0b641a51432f9172ed0b41521822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5563a27e4e4748813d624aab13cdca in term 1.
I20250830 01:58:50.595338 27727 leader_election.cc:304] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d61552c37a44ad9a7a6f6f5c236227, 7f5563a27e4e4748813d624aab13cdca; no voters:
I20250830 01:58:50.595690 29253 raft_consensus.cc:2802] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 1 FOLLOWER]: Leader election won for term 1
I20250830 01:58:50.595806 29253 raft_consensus.cc:695] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [term 1 LEADER]: Becoming Leader. State: Replica: 7f5563a27e4e4748813d624aab13cdca, State: Running, Role: LEADER
I20250830 01:58:50.596097 29253 consensus_queue.cc:237] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 } } peers { permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" member_type: VOTER last_known_addr { host: "127.26.105.194" port: 43787 } } peers { permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 } }
I20250830 01:58:50.614670 29409 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ad9824f7e6604077ac8a034879b4c731)
I20250830 01:58:50.614871 29409 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:50.615634 29409 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:50.617477 29306 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:50.617643 29307 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:50.617813 29307 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:50.618057 29409 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:58:50.858134 27047 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu pid 29283
I20250830 01:58:50.865358 29350 generic_service.cc:196] Checking for leaks (request via RPC)
I20250830 01:58:50.973088 27047 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu with pid 29283
I20250830 01:58:51.204288 29253 consensus_queue.cc:1035] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:51.209367 29469 consensus_queue.cc:1035] T 320d5ead70554ba6a39212ea95a6ffb3 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:51.471449 29466 consensus_queue.cc:1035] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:51.476084 29253 consensus_queue.cc:1035] T 2b91ee2988c84569b779ea9292dcaad6 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:51.499689 29253 consensus_queue.cc:1035] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "893b0b641a51432f9172ed0b41521822" member_type: VOTER last_known_addr { host: "127.26.105.195" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:51.504091 29469 consensus_queue.cc:1035] T dcd2a38e964f4d5fab198259e0686378 P 7f5563a27e4e4748813d624aab13cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" member_type: VOTER last_known_addr { host: "127.26.105.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250830 01:58:51.880926 27047 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
/tmp/dist-test-taskqeOeTE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--webserver_interface=127.26.105.252
--webserver_port=39345
--builtin_ntp_servers=127.26.105.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250830 01:58:52.217938 29484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250830 01:58:52.218309 29484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250830 01:58:52.218449 29484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250830 01:58:52.226761 29484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250830 01:58:52.226858 29484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250830 01:58:52.226922 29484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250830 01:58:52.226989 29484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250830 01:58:52.236732 29484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.105.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43413
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.105.252:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.105.252
--webserver_port=39345
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 30 Aug 2025 01:43:21 UTC on 24a791456cd2
build id 7808
ASAN enabled
I20250830 01:58:52.238364 29484 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250830 01:58:52.240137 29484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250830 01:58:52.248165 29497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:52.248521 29499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250830 01:58:52.248319 29496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250830 01:58:52.249464 29484 server_base.cc:1047] running on GCE node
I20250830 01:58:52.250238 29484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250830 01:58:52.251578 29484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250830 01:58:52.252779 29484 hybrid_clock.cc:648] HybridClock initialized: now 1756519132252742 us; error 27 us; skew 500 ppm
I20250830 01:58:52.253165 29484 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250830 01:58:52.255369 29484 webserver.cc:480] Webserver started at http://127.26.105.252:39345/ using document root <none> and password file <none>
I20250830 01:58:52.256069 29484 fs_manager.cc:362] Metadata directory not provided
I20250830 01:58:52.256181 29484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250830 01:58:52.261600 29484 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20250830 01:58:52.264364 29505 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250830 01:58:52.265753 29484 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250830 01:58:52.265913 29484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data,/tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
uuid: "fd4f5c1e7015460e99715d8a2edd2959"
format_stamp: "Formatted at 2025-08-30 01:58:20 on dist-test-slave-gp2r"
I20250830 01:58:52.266637 29484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250830 01:58:52.299293 29484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250830 01:58:52.300459 29484 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250830 01:58:52.300761 29484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250830 01:58:52.321384 29484 rpc_server.cc:307] RPC server started. Bound to: 127.26.105.252:46233
I20250830 01:58:52.321408 29556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.105.252:46233 every 8 connection(s)
I20250830 01:58:52.323053 29484 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqeOeTE/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756519082367942-27047-0/minicluster-data/master-2/data/info.pb
I20250830 01:58:52.329540 29557 sys_catalog.cc:263] Verifying existing consensus state
I20250830 01:58:52.330541 27047 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqeOeTE/build/asan/bin/kudu as pid 29484
I20250830 01:58:52.333801 29557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap starting.
I20250830 01:58:52.345413 29557 log.cc:826] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Log is configured to *not* fsync() on all Append() calls
I20250830 01:58:52.352880 27974 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:52.374976 29557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=19 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250830 01:58:52.375634 29557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: Bootstrap complete.
I20250830 01:58:52.380658 29557 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:52.381280 29557 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4f5c1e7015460e99715d8a2edd2959, State: Initialized, Role: FOLLOWER
I20250830 01:58:52.381891 29557 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 8.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:52.382714 29565 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } } }
I20250830 01:58:52.383071 29565 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: This master's current role is: FOLLOWER
I20250830 01:58:52.383930 29557 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [sys.catalog]: configured and running, proceeding with master startup.
I20250830 01:58:52.392652 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:58:52.398142 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:58:52.398190 29579 catalog_manager.cc:1261] Loaded cluster ID: b3f54d3a81d9420093a5097ef44cfe8f
I20250830 01:58:52.398339 29579 catalog_manager.cc:1554] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: loading cluster ID for follower catalog manager: success
I20250830 01:58:52.401266 29579 catalog_manager.cc:1576] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: acquiring CA information for follower catalog manager: success
I20250830 01:58:52.404081 29579 catalog_manager.cc:1604] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250830 01:58:52.501803 27974 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:52.503504 27838 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:52.504590 29522 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "893b0b641a51432f9172ed0b41521822" instance_seqno: 1756519102177221) as {username='slave'} at 127.26.105.195:48339; Asking this server to re-register.
I20250830 01:58:52.505277 29521 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f5563a27e4e4748813d624aab13cdca" instance_seqno: 1756519101628547) as {username='slave'} at 127.26.105.194:45597; Asking this server to re-register.
I20250830 01:58:52.505338 27974 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:52.505817 27838 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:52.507133 29521 ts_manager.cc:194] Registered new tserver with Master: 893b0b641a51432f9172ed0b41521822 (127.26.105.195:35297)
I20250830 01:58:52.507280 29522 ts_manager.cc:194] Registered new tserver with Master: 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787)
I20250830 01:58:52.517058 27699 heartbeater.cc:344] Connected to a master server at 127.26.105.252:46233
I20250830 01:58:52.518695 29522 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "03d61552c37a44ad9a7a6f6f5c236227" instance_seqno: 1756519101099193) as {username='slave'} at 127.26.105.193:54361; Asking this server to re-register.
I20250830 01:58:52.519296 27699 heartbeater.cc:461] Registering TS with master...
I20250830 01:58:52.520542 29522 ts_manager.cc:194] Registered new tserver with Master: 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219)
I20250830 01:58:52.794060 29565 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:52.794417 29565 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:52.796206 29565 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:52.796957 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:52.798722 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:52.798952 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:52.799060 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:52.799463 29565 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:58:53.308018 29593 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:53.308189 29593 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:53.308881 29593 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:53.312074 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:58:53.312288 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:58:53.312418 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:53.312675 29593 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:58:53.894953 29602 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:53.895166 29602 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:53.895889 29602 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:53.898023 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:53.898313 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:53.898504 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:53.898834 29602 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:58:54.470206 27047 master-stress-test.cc:577] Killing test thread 1/25
I20250830 01:58:54.894666 29621 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:54.894883 29621 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:54.895669 29621 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:54.897773 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:54.897917 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:54.898072 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:54.898352 29621 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:58:56.854322 29652 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:56.854558 29652 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:56.855217 29652 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:56.857388 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:56.857553 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:56.857705 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:56.857949 29652 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:58:57.402045 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:58:57.402307 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:58:58.450328 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 16: this message will repeat every 5th retry.
I20250830 01:58:58.498366 29681 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:58:58.498590 29681 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:58:58.499248 29681 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:58:58.500236 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250830 01:58:58.501351 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:58:58.501502 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:58:58.501619 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:58:58.501852 29681 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 01:58:59.816103 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 16: this message will repeat every 5th retry.
I20250830 01:59:02.250936 29743 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:59:02.251132 29743 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:59:02.251755 29743 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:59:02.253830 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:59:02.253990 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:59:02.254110 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:59:02.254397 29743 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:59:02.403556 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:02.403800 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:59:07.405289 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:07.405514 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:59:07.525467 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 21: this message will repeat every 5th retry.
W20250830 01:59:09.857553 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 21: this message will repeat every 5th retry.
I20250830 01:59:10.486622 29839 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:59:10.486807 29839 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:59:10.487561 29839 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:59:10.488529 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250830 01:59:10.489749 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:59:10.489923 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:59:10.490046 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:59:10.490327 29839 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:59:12.407020 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:12.407195 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:59:14.675362 29894 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:59:14.675559 29894 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:59:14.676242 29894 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:59:14.678432 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:59:14.678575 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:59:14.678717 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:59:14.678961 29894 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:59:17.408785 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:17.409010 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:59:17.768322 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 26: this message will repeat every 5th retry.
W20250830 01:59:19.618045 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 26: this message will repeat every 5th retry.
I20250830 01:59:22.410548 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:22.410722 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:59:23.084733 27700 heartbeater.cc:646] Failed to heartbeat to 127.26.105.254:36635 (71 consecutive failures): Network error: Failed to ping master at 127.26.105.254:36635: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:59:23.089946 27839 heartbeater.cc:646] Failed to heartbeat to 127.26.105.254:36635 (73 consecutive failures): Network error: Failed to ping master at 127.26.105.254:36635: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:59:23.112082 27976 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (126 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:59:27.412449 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:27.412664 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:59:28.843778 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 31: this message will repeat every 5th retry.
I20250830 01:59:29.297406 30055 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:59:29.297601 30055 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:59:29.298404 30055 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:59:29.299405 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250830 01:59:29.300891 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 01:59:29.301247 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 01:59:29.301419 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:59:29.301679 30055 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 01:59:29.861712 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 31: this message will repeat every 5th retry.
I20250830 01:59:32.413998 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:32.414208 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:59:37.415628 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:37.415817 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:59:38.334131 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 36: this message will repeat every 5th retry.
W20250830 01:59:39.692224 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 36: this message will repeat every 5th retry.
I20250830 01:59:42.417289 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:42.417479 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:59:44.781009 30224 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 01:59:44.781934 30224 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 01:59:44.782826 30224 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 01:59:44.783891 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 01:59:44.785267 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 01:59:44.785416 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 01:59:44.785666 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 01:59:44.785907 30224 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 01:59:47.418846 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:47.419031 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:59:47.811553 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 41: this message will repeat every 5th retry.
W20250830 01:59:50.021090 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 41: this message will repeat every 5th retry.
I20250830 01:59:52.420480 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:52.420666 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 01:59:57.422451 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 01:59:57.422658 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 01:59:57.938143 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 46: this message will repeat every 5th retry.
W20250830 02:00:00.312186 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 46: this message will repeat every 5th retry.
I20250830 02:00:02.424881 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:02.425094 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:00:03.575819 30458 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:00:03.576053 30458 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:00:03.577411 30458 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:00:03.578469 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:00:03.583389 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:00:03.583568 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:00:03.583686 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:00:03.583909 30458 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:00:07.426626 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:07.426802 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:00:07.729733 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 51: this message will repeat every 5th retry.
W20250830 02:00:10.021670 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 51: this message will repeat every 5th retry.
I20250830 02:00:12.428680 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:12.428872 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:00:17.430935 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:17.431143 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:00:17.431603 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 56: this message will repeat every 5th retry.
W20250830 02:00:20.254191 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 56: this message will repeat every 5th retry.
I20250830 02:00:21.023792 30730 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:00:21.024026 30730 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:00:21.024863 30730 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:00:21.026990 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:00:21.028259 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:00:21.028633 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:00:21.028831 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:00:21.029222 30730 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:00:22.290722 30760 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:00:22.290923 30760 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:00:22.291496 30760 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:00:22.294333 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:00:22.294523 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:00:22.294664 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:00:22.294924 30760 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:00:22.432281 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:22.432624 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:00:23.127471 27976 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (307 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:00:23.246757 27701 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (204 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:00:23.272598 27842 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (207 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:00:27.299345 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 61: this message will repeat every 5th retry.
I20250830 02:00:27.434437 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:27.434645 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:00:29.960274 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 61: this message will repeat every 5th retry.
I20250830 02:00:32.436180 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:32.436388 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:00:37.437999 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:37.438326 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:00:37.775823 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 66: this message will repeat every 5th retry.
W20250830 02:00:39.943851 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 66: this message will repeat every 5th retry.
I20250830 02:00:40.286784 31030 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:00:40.286983 31030 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:00:40.287854 31030 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:00:40.288874 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250830 02:00:40.290241 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:00:40.290457 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:00:40.290585 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:00:40.290854 31030 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:00:42.434763 31069 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:00:42.434938 31069 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:00:42.435539 31069 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:00:42.437495 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:00:42.437669 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:00:42.437776 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:00:42.437992 31069 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:00:42.439078 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:42.439242 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:00:47.248952 31140 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:00:47.249190 31140 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:00:47.250097 31140 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:00:47.251052 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250830 02:00:47.252274 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:00:47.252492 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:00:47.252610 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:00:47.252866 31140 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:00:47.440339 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:47.440536 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:00:48.410957 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 71: this message will repeat every 5th retry.
W20250830 02:00:50.739209 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 71: this message will repeat every 5th retry.
I20250830 02:00:52.441946 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:52.442112 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:00:52.469707 29511 maintenance_manager.cc:419] P fd4f5c1e7015460e99715d8a2edd2959: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033357
I20250830 02:00:52.519008 29510 maintenance_manager.cc:643] P fd4f5c1e7015460e99715d8a2edd2959: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.048s user 0.018s sys 0.011s Metrics: {"bytes_written":11164,"cfile_init":1,"dirs.queue_time_us":352,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":934,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":795,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":19,"thread_start_us":516,"threads_started":2,"wal-append.queue_time_us":388}
I20250830 02:00:52.520191 29511 maintenance_manager.cc:419] P fd4f5c1e7015460e99715d8a2edd2959: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1056 bytes on disk
I20250830 02:00:52.521065 29510 maintenance_manager.cc:643] P fd4f5c1e7015460e99715d8a2edd2959: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20250830 02:00:55.139878 31271 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:00:55.140084 31271 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:00:55.140908 31271 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:00:55.141916 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:00:55.143062 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:00:55.143280 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:00:55.143415 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:00:55.143687 31271 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:00:57.443403 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:00:57.443575 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:00:58.395227 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 76: this message will repeat every 5th retry.
W20250830 02:01:01.648610 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 76: this message will repeat every 5th retry.
I20250830 02:01:02.445060 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:02.445246 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:01:07.446743 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:07.446938 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:01:07.817785 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 81: this message will repeat every 5th retry.
I20250830 02:01:12.329200 31602 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:01:12.329409 31602 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:01:12.330250 31602 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:01:12.331323 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:01:12.332628 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:01:12.332818 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:01:12.332942 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:01:12.333186 31602 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:01:12.448143 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:12.448343 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:01:12.457305 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 81: this message will repeat every 5th retry.
I20250830 02:01:13.479773 31624 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:01:13.479988 31624 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:01:13.480614 31624 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:01:13.482807 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:01:13.482990 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:01:13.483120 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:01:13.483381 31624 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:01:16.204003 31678 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:01:16.204197 31678 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:01:16.204788 31678 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:01:16.206983 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:01:16.207167 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:01:16.207281 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:01:16.207530 31678 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:01:17.449788 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:17.449982 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:01:17.532888 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 86: this message will repeat every 5th retry.
W20250830 02:01:22.351379 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 86: this message will repeat every 5th retry.
I20250830 02:01:22.451404 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:22.451594 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:01:23.201376 27975 heartbeater.cc:646] Failed to heartbeat to 127.26.105.254:36635 (473 consecutive failures): Network error: Failed to ping master at 127.26.105.254:36635: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:01:23.429636 27701 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (324 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:01:23.458067 27842 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (327 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:01:27.022387 31881 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:01:27.022590 31881 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:01:27.023253 31881 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:01:27.024259 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250830 02:01:27.025415 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:01:27.026216 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:01:27.026335 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:01:27.026611 31881 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:01:27.452864 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:27.453115 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:01:27.635217 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 91: this message will repeat every 5th retry.
W20250830 02:01:32.139293 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 91: this message will repeat every 5th retry.
I20250830 02:01:32.454919 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:32.455668 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:01:37.457656 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:37.457859 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:01:37.840415 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 96: this message will repeat every 5th retry.
I20250830 02:01:38.502360 32085 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:01:38.502571 32085 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:01:38.503890 32085 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:01:38.504868 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:01:38.506912 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:01:38.507146 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:01:38.507289 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:01:38.507534 32085 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 02:01:42.278460 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 96: this message will repeat every 5th retry.
I20250830 02:01:42.459559 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:42.459758 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:01:47.461174 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:47.461350 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:01:47.703325 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 101: this message will repeat every 5th retry.
W20250830 02:01:52.034428 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 101: this message will repeat every 5th retry.
I20250830 02:01:52.463035 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:52.463240 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:01:57.464884 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:01:57.465121 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:01:58.085405 32454 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:01:58.085583 32454 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:01:58.086548 32454 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:01:58.087754 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:01:58.089267 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:01:58.089495 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:01:58.089594 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:01:58.089897 32454 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 02:01:58.193138 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 106: this message will repeat every 5th retry.
W20250830 02:02:02.072819 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 106: this message will repeat every 5th retry.
I20250830 02:02:02.376686 32533 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:02:02.376873 32533 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:02:02.377441 32533 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:02:02.379966 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:02:02.380151 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:02:02.380254 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:02:02.380507 32533 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:02:02.466538 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:02.466784 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:02:07.468434 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:07.468669 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:02:07.950659 32636 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:02:07.950862 32636 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:02:07.951562 32636 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:02:07.952149 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250830 02:02:07.953485 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:02:07.953646 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:02:07.953773 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:02:07.954043 32636 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 02:02:08.332607 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 111: this message will repeat every 5th retry.
W20250830 02:02:10.916574 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 111: this message will repeat every 5th retry.
I20250830 02:02:11.055049 32695 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:02:11.055243 32695 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:02:11.056422 32695 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:02:11.058418 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:02:11.058598 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:02:11.058740 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:02:11.059005 32695 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:02:12.469996 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:12.470238 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:02:17.471804 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:17.472057 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:02:18.126677 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 116: this message will repeat every 5th retry.
W20250830 02:02:20.163199 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 116: this message will repeat every 5th retry.
I20250830 02:02:22.473510 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:22.473711 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:02:22.671432 435 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:02:22.671631 435 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:02:22.672407 435 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:02:22.673163 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250830 02:02:22.675057 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:02:22.675259 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:02:22.675384 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:02:22.675621 435 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 02:02:23.613991 27701 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (444 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:02:23.646589 27839 heartbeater.cc:646] Failed to heartbeat to 127.26.105.254:36635 (433 consecutive failures): Network error: Failed to ping master at 127.26.105.254:36635: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:02:23.699924 27975 heartbeater.cc:646] Failed to heartbeat to 127.26.105.254:36635 (656 consecutive failures): Network error: Failed to ping master at 127.26.105.254:36635: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:02:27.475155 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:27.475360 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:02:28.193982 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 121: this message will repeat every 5th retry.
W20250830 02:02:29.584321 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 121: this message will repeat every 5th retry.
I20250830 02:02:30.063560 563 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:02:30.063802 563 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:02:30.064445 563 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:02:30.065451 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:02:30.066985 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:02:30.067138 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:02:30.067256 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:02:30.067502 563 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:02:32.476851 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:32.477036 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:02:33.784531 642 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:02:33.785058 642 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:02:33.786063 642 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:02:33.789047 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:02:33.789301 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:02:33.789414 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:02:33.789710 642 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:02:37.479008 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:37.479244 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:02:38.320156 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 126: this message will repeat every 5th retry.
W20250830 02:02:40.065375 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 126: this message will repeat every 5th retry.
I20250830 02:02:42.480633 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:42.480861 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:02:47.482669 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:47.482905 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:02:48.964783 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 131: this message will repeat every 5th retry.
I20250830 02:02:49.210870 928 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:02:49.211081 928 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:02:49.211916 928 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:02:49.213011 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250830 02:02:49.214401 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:02:49.214550 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:02:49.214711 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:02:49.215006 928 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 02:02:50.016444 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 131: this message will repeat every 5th retry.
I20250830 02:02:52.484248 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:52.484475 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:02:57.485977 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:02:57.486209 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:02:58.523825 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 136: this message will repeat every 5th retry.
W20250830 02:03:00.328994 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 136: this message will repeat every 5th retry.
I20250830 02:03:01.999049 1170 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:03:01.999269 1170 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:03:02.000030 1170 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:03:02.001055 29508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:03:02.002274 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:03:02.002539 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:03:02.002671 29509 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:03:02.002874 1170 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:03:02.487382 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:02.487607 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:03:03.259897 1193 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:03:03.260072 1193 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:03:03.260699 1193 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:03:03.262868 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:03.263051 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:03:03.263170 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:03:03.263410 1193 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:03:04.917150 1231 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:03:04.917361 1231 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:03:04.918061 1231 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:03:04.920404 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:04.920606 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:03:04.920717 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:03:04.920969 1231 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:03:07.489013 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:07.489236 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:03:08.423287 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 141: this message will repeat every 5th retry.
I20250830 02:03:08.468232 1298 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:03:08.468464 1298 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:03:08.469194 1298 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:03:08.470211 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250830 02:03:08.471459 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:08.471628 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:03:08.471755 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:03:08.472040 1298 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250830 02:03:10.085536 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 141: this message will repeat every 5th retry.
I20250830 02:03:12.490685 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:12.490872 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:03:17.492221 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:17.492401 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:03:18.691166 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 146: this message will repeat every 5th retry.
W20250830 02:03:20.248355 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 146: this message will repeat every 5th retry.
I20250830 02:03:22.493917 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:22.494102 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:03:23.796000 27700 heartbeater.cc:646] Failed to heartbeat to 127.26.105.254:36635 (551 consecutive failures): Network error: Failed to ping master at 127.26.105.254:36635: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:23.826239 27842 heartbeater.cc:646] Failed to heartbeat to 127.26.105.253:39521 (567 consecutive failures): Network error: Failed to ping master at 127.26.105.253:39521: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
W20250830 02:03:23.873333 27975 heartbeater.cc:646] Failed to heartbeat to 127.26.105.254:36635 (843 consecutive failures): Network error: Failed to ping master at 127.26.105.254:36635: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
I20250830 02:03:24.850498 1597 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:03:24.850701 1597 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:03:24.851934 1597 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:03:24.852885 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:03:24.855000 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:24.856215 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:03:24.856326 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:03:24.857071 1597 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:03:27.496317 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:27.496578 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:03:29.427743 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 151: this message will repeat every 5th retry.
W20250830 02:03:30.239998 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 151: this message will repeat every 5th retry.
I20250830 02:03:32.498070 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:32.498301 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:03:32.648504 1720 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:03:32.648699 1720 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:03:32.649397 1720 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:03:32.649941 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:03:32.651185 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:32.651432 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:03:32.651544 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:03:32.651798 1720 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:03:37.499794 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:37.500010 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:03:39.389420 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 156: this message will repeat every 5th retry.
W20250830 02:03:40.710652 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 156: this message will repeat every 5th retry.
I20250830 02:03:42.501382 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:42.501595 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250830 02:03:42.532447 1877 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250830 02:03:42.532644 1877 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9824f7e6604077ac8a034879b4c731" member_type: VOTER last_known_addr { host: "127.26.105.254" port: 36635 } } peers { permanent_uuid: "9ba697035f304124a88a1cf1aabb748e" member_type: VOTER last_known_addr { host: "127.26.105.253" port: 39521 } } peers { permanent_uuid: "fd4f5c1e7015460e99715d8a2edd2959" member_type: VOTER last_known_addr { host: "127.26.105.252" port: 46233 } }
I20250830 02:03:42.533994 1877 leader_election.cc:290] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635), 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521)
W20250830 02:03:42.534539 29509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250830 02:03:42.536114 29509 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad9824f7e6604077ac8a034879b4c731 (127.26.105.254:36635): Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:42.536301 29508 leader_election.cc:336] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9ba697035f304124a88a1cf1aabb748e (127.26.105.253:39521): Network error: Client connection negotiation failed: client connection to 127.26.105.253:39521: connect: Connection refused (error 111)
I20250830 02:03:42.536422 29508 leader_election.cc:304] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4f5c1e7015460e99715d8a2edd2959; no voters: 9ba697035f304124a88a1cf1aabb748e, ad9824f7e6604077ac8a034879b4c731
I20250830 02:03:42.536635 1877 raft_consensus.cc:2747] T 00000000000000000000000000000000 P fd4f5c1e7015460e99715d8a2edd2959 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250830 02:03:47.503165 29484 master_runner.cc:186] Error getting master registration for 127.26.105.254:36635: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111),
I20250830 02:03:47.503340 29484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250830 02:03:50.128957 27863 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 03d61552c37a44ad9a7a6f6f5c236227 (127.26.105.193:36219): Couldn't send request to peer 03d61552c37a44ad9a7a6f6f5c236227. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 161: this message will repeat every 5th retry.
I20250830 02:03:50.279057 29511 maintenance_manager.cc:419] P fd4f5c1e7015460e99715d8a2edd2959: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1056 bytes on disk
I20250830 02:03:50.285147 29510 maintenance_manager.cc:643] P fd4f5c1e7015460e99715d8a2edd2959: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.001s Metrics: {"dirs.queue_time_us":426,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":91,"thread_start_us":505,"threads_started":2,"wal-append.queue_time_us":339}
W20250830 02:03:50.730732 27864 consensus_peers.cc:489] T cddb4c87468a44d1a2bae0fbd6784a36 P 893b0b641a51432f9172ed0b41521822 -> Peer 7f5563a27e4e4748813d624aab13cdca (127.26.105.194:43787): Couldn't send request to peer 7f5563a27e4e4748813d624aab13cdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cddb4c87468a44d1a2bae0fbd6784a36. This is attempt 161: this message will repeat every 5th retry.
W20250830 02:03:50.972396 27984 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:50.972687 27984 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
W20250830 02:03:50.972838 27984 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
F20250830 02:03:50.983745 27999 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_badff657ae07402a88632f78ec1befb7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1756519430 (unix time) try "date -d @1756519430" if you are using GNU date ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989573 28000 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989573 28000 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989611 28011 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000069a7) received by PID 27047 (TID 0x7ff5175ab700) from PID 27047; stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989573 28000 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989611 28011 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.990927 27994 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_509af6139e2e4ba6bc5c537af315df6a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989573 28000 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989611 28011 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.990927 27994 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_509af6139e2e4ba6bc5c537af315df6a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992044 28015 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992076 2*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989573 28000 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989611 28011 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.990927 27994 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_509af6139e2e4ba6bc5c537af315df6a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992044 28015 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992076 28002 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7ff52e730980 (unknown) at ??:0
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989573 28000 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989611 28011 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.990927 27994 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_509af6139e2e4ba6bc5c537af315df6a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992044 28015 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992076 28002 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.993355 28003 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250830 02:03:50.984851 28014 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985064 27997 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11641b1a96984fa099ffa232b6a0033e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985244 27995 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a36b1b6bfc854dfc907635b41d47be02 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.985282 28016 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986362 27990 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_739ceee58b144d72b56342855499ecc4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.986490 27996 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_624ff0828e534d54a8ea5f69551f7e8f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987637 27992 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1ad78e45bc5842aeaf69a74b3ef38007 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.987656 27991 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4634c7907fc46558619aee806e21da3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.988888 27998 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac070120b97141e188efab1e24ece67a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989074 28001 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989538 28010 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989573 28000 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.989611 28011 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.990927 27994 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_509af6139e2e4ba6bc5c537af315df6a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992044 28015 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.992076 28002 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.993355 28003 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)F20250830 02:03:50.993480 27993 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cef396d7e593458aa7679269acecf621 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.26.105.254:36635,127.26.105.253:39521,127.26.105.252:46233, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.26.105.254:36635: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7ff527704fb7 gsignal at ??:0
@ 0x7ff527706921 abort at ??:0
@ 0x7ff52a43cdcd google::LogMessage::Fail() at ??:0
@ 0x7ff52a440b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff52a43c7cc google::LogMessage::Flush() at ??:0
@ 0x7ff52a43df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b582 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7ff527d8c6df (unknown) at ??:0
@ 0x7ff52e7256db start_thread at ??:0
@ 0x7ff5277e771f clone at ??:0